Nov 02 14:55:33.981988 ip-10-0-129-45 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:55:34.381522 ip-10-0-129-45 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:34.381522 ip-10-0-129-45 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:55:34.381522 ip-10-0-129-45 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:34.381522 ip-10-0-129-45 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:55:34.386378 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386281    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.386378 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386375    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.386378 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386379    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.386378 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386383    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.386378 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386386    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386390    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386393    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386396    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386399    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386402    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386404    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386407    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386410    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386413    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386416    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386419    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386422    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386424    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386427    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386430    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386432    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386435    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386438    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.386579 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386441    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386444    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386447    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386451    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386456    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386460    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386464    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386468    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386472    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386477    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386481    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386485    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386489    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386493    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386498    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386503    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386508    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386512    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386517    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386521    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.387045 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386526    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386531    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386536    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386541    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386546    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386550    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386555    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386560    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386565    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386570    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386574    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386579    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386584    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386590    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386598    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386606    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386611    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386616    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386620    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.387550 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386623    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386627    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386631    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386636    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386640    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386659    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386663    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386666    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386669    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386672    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386675    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386677    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386680    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386683    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386685    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386688    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386691    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386694    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386697    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.388021 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386700    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386702    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386706    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386709    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386711    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386714    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386717    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386720    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386723    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386726    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386729    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386731    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386734    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386738    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386742    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386745    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386747    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386750    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386755    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.388470 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386758    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386761    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386764    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386767    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386770    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.386773    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387573    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387580    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387583    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387585    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387589    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387592    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387595    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387598    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387600    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387603    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387606    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387609    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387612    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387615    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.388949 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387618    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387621    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387624    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387626    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387629    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387632    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387634    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387637    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387640    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387642    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387660    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387664    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387667    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387670    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387672    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387677    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387681    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387684    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387686    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.389427 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387689    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387692    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387695    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387698    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387701    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387703    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387706    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387709    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387711    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387714    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387717    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387720    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387723    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387726    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387728    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387731    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387733    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387736    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387739    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.389902 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387741    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387744    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387746    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387749    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387752    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387755    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387758    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387761    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387764    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387766    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387769    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387772    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387775    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387777    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387780    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387783    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387786    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387788    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387791    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.390359 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387794    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387796    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387799    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387802    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387804    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387807    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387813    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387819    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387822    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387826    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387829    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387831    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387834    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387837    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387840    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387842    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387845    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387848    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387851    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387854    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.390818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387856    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387859    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387862    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387864    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387867    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387869    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387872    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387875    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387878    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387880    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387883    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387885    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387888    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387891    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.387893    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388381    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388386    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388390    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388394    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388397    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388400    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.391289 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388403    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388406    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388408    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388411    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388413    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388416    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388419    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388422    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388424    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388427    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388429    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388432    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388435    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388437    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388440    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388443    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388446    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388448    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388451    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388453    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.391818 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388458    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388460    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388463    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388466    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388468    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388471    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388474    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388477    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388479    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388483    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388486    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388489    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388493    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388497    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388500    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388503    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388506    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388509    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388512    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.392280 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388514    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388516    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388519    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388522    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388524    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388527    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388529    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388532    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388535    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388537    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388540    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388542    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388545    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388549    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388552    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388554    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388557    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388559    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388562    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388565    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.392783 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388567    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388570    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388573    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388576    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388578    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388581    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388584    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388586    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388589    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388591    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388594    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388596    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388600    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388602    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388605    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388608    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388610    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388614    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388616    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.393266 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388619    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388621    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388624    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388627    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388629    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388632    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388635    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388638    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388641    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388657    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388660    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388663    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388667    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388671    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388675    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388678    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388681    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388685    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388689    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.393734 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388692    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388696    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.388699    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388785    2581 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388793    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388800    2581 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388804    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388809    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388812    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388819    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388824    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388827    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388830    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388834    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388837    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388840    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388843    2581 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388846    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388849    2581 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388854    2581 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388857    2581 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388860    2581 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388863    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:55:34.394216 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388866    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388870    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388873    2581 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388877    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388880    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388885    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388888    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388892    2581 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388895    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388898    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388902    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388905    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388909    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388912    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388915    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388918    2581 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388922    2581 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388925    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388930    2581 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388933    2581 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388936    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388939    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388942    2581 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388946    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:55:34.394763 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388949    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388953    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388956    2581 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388960    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388964    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388967    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388970    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388972    2581 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388975    2581 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388979    2581 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388983    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388986    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388989    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388994    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.388997    2581 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389000    2581 flags.go:64] FLAG: --help="false"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389003    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389007    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389010    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389013    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389017    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389020    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389023    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:55:34.395362 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389026    2581 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389029    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389032    2581 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389035    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389039    2581 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389042    2581 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389045    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389048    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389051    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389054    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389057    2581 flags.go:64] FLAG: --lock-file=""
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389060    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389063    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389067    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389072    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389075    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389078    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389081    2581 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389084    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389088    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389091    2581 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389094    2581 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389100    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389103    2581 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389107    2581 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:55:34.395953 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389110    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389113    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389116    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389119    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389123    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389126    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389129    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389136    2581 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389140    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389143    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389146    2581 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389149    2581 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389152    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389155    2581 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389158    2581 flags.go:64] FLAG: --port="10250"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389162    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389164    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bb0bfb19f65ff53e"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389168    2581 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389171    2581 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389174    2581 flags.go:64] FLAG: --register-node="true"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389177    2581 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389181    2581 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389184    2581 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389187    2581 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389189    2581 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389194    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:55:34.396538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389197    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389200    2581 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389203    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389208    2581 flags.go:64] FLAG: --runonce="false"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389211    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389214    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389217    2581 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389220    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389224    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389227    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389231    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389234    2581 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389237    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389240    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389243    2581 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389246    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389250    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389253    2581 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389256    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389261    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389264    2581 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389267    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389271    2581 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389273    2581 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389276    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:55:34.397172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389279    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389282    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389285    2581 flags.go:64] FLAG: --v="2"
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389290    2581 flags.go:64] FLAG: --version="false"
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389294    2581 flags.go:64] FLAG: --vmodule=""
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389297    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389301    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389427    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389431    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389434    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389438    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389441    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389444    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389447    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389449    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389452    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389455    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389458    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389462    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389465    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389468    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.397795 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389470    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389473    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389475    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389478    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389481    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389484    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389486    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389489    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389491    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389494    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389497    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389499    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389502    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389505    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389507    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389510    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389513    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389516    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389518    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389521    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.398286 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389524    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389528    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389532    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389534    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389537    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389541    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389545    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389548    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389551    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389555    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389558    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389561    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389563    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389566    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389568    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389571    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389574    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389576    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389579    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.398769 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389582    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389585    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389587    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389590    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389592    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389597    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389601    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389604    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389607    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389610    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389612    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389615    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389618    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389620    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389625    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389627    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389630    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389633    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389635    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389638    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.399219 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389641    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389656    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389660    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389664    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389667    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389669    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389672    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389675    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389678    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389681    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389684    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389686    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389689    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389692    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389695    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389697    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389700    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389703    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389706    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.399793 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389708    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389711    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389714    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389716    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389719    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389722    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389724    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389728    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389731    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389734    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389736    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389739    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389741    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389744    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.389769    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389807    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.400244 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389811    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389815    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389818    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389821    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389824    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389826    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389829    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389832    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389834    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389837    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389839    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389842    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389845    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389848    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389850    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389853    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389856    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389859    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389861    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389864    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.400620 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389866    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389870    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389872    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389875    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389879    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389881    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389884    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389887    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389890    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389892    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389895    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389899    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389901    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389905    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389908    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389911    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389914    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389917    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389919    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.401140 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389922    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389925    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389927    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389930    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389932    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389935    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389938    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389941    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389943    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389946    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389949    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389951    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389954    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389957    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389959    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389962    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389965    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389969    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389972    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.401631 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389974    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389977    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389979    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389982    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389985    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389988    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389991    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389994    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389996    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.389999    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390002    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390005    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390007    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390010    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390013    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390015    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390018    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390020    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390023    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390026    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.402138 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390028    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390031    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390034    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390037    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390040    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390043    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390047    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390050    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390052    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390056    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390060    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390063    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390065    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390068    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390071    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390073    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390076    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390080    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390083    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.402605 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390086    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390088    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390091    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390094    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390097    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390101    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390105    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.390108    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.397445    2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.397461    2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.397475    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.397480    2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397540    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397546    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397549    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397552    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397555    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397557    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397560    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.403132 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397563    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397565    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397568    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397571    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397573    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397576    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397579    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397582    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397585    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397587    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397590    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397593    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397595    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397598    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397601    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397603    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397606    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397608    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397611    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397614    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.403617 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397616    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397619    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397629    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397632    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397635    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397637    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397642    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397657    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397661    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397665    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397667    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397670    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397673    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397676    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397678    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397681    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397684    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397687    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397690    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.404167 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397693    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397696    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397698    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397701    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397703    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397706    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397709    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397711    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397714    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397717    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397721    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397725    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397728    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397731    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397734    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397737    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397747    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397750    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397752    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.404745 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397755    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397758    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397760    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397763    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397766    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397769    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397771    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397774    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397777    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397780    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397782    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397785    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397788    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397790    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397793    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397796    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397799    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397802    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397805    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397807    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.405490 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397810    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397813    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397815    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397818    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397821    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397823    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397826    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397829    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397831    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397834    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397845    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397848    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397851    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397853    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397856    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397859    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397861    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397864    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397867    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.406348 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397870    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.397872    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.397903    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398106    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398111    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398114    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398116    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398119    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398122    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398124    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398127    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398130    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398132    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398137    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398141    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.406816 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398144    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398147    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398150    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398153    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398155    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398158    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398161    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398164    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398167    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398175    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398178    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398181    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398183    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398186    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398189    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398192    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398195    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398197    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398200    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398203    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.407164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398205    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398208    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398211    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398213    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398216    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398218    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398221    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398224    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398226    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398229    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398231    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398234    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398236    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398239    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398242    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398244    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398247    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398250    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398252    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.407660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398255    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398257    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398260    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398268    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398271    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398273    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398277    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398280    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398282    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398285    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398287    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398290    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398293    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398296    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398298    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398301    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398304    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398306    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398309    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.408098 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398312    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398314    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398317    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398320    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398323    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398325    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398334    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398336    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398339    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398348    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398351    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398354    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398356    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398359    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398362    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398364    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398367    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398375    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398379    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398383    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.408660 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398387    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398390    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398393    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398396    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398399    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398402    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398404    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398407    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398409    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398412    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398415    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398417    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398420    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398422    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398425    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398428    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.409336 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.398452    2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398489    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398493    2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398496    2581 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398499    2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398501    2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398504    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398506    2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398509    2581 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398512    2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398514    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398517    2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398519    2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398522    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398530    2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398533    2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398535    2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398538    2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398540    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398543    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:55:34.409829 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398546    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398548    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398551    2581 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398553    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398556    2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398558    2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398561    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398564    2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398566    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398569    2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398571    2581 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398574    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398576    2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398579    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398582    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398584    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398587    2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398589    2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398592    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:55:34.410291 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398595    2581 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398597    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398600    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398603    2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398605    2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398608    2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398610    2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398613    2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398627    2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398630    2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398632    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398635    2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398637    2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398640    2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398643    2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398656    2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398658    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398661    2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398664    2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:55:34.410750 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398667    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398670    2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398673    2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398675    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398678    2581 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398680    2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398683    2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398686    2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398688    2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398691    2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398693    2581 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398696    2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398698    2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398701    2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398704    2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398708    2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398711    2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398715    2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398718    2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398721    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:55:34.411223 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398723    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398726    2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398735    2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398737    2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398740    2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398742    2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398745    2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398747    2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398750    2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398752    2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398755    2581 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398760    2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398762    2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398765    2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398768    2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398771    2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398774    2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398777    2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398781    2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398784    2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:55:34.411706 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398787    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398789    2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398792    2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398796    2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398800    2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398803    2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398806    2581 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398814    2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:34.398816    2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.398876    2581 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.401053    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.401847    2581 server.go:1014] "Starting client certificate rotation"
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.401959    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.403052    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.404612    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:34.412162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.411635    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:55:34.420708 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.420694    2581 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:55:34.423802 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.423789    2581 log.go:25] "Validated CRI v1 image API"
Nov 02 14:55:34.424568 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.424542    2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:55:34.428916 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.428895    2581 fs.go:135] Filesystem UUIDs: map[4e612a45-5c89-46eb-abc7-a50bd3708726:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9e14c4db-ebfb-4212-a030-72a6c932702b:/dev/nvme0n1p3]
Nov 02 14:55:34.428985 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.428914    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 14:55:34.434203 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.434091    2581 manager.go:217] Machine: {Timestamp:2026-11-02 14:55:34.432803459 +0000 UTC m=+0.326686704 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3087007 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f2069002d9bc5b7e024f8336c08d SystemUUID:ec22f206-9002-d9bc-5b7e-024f8336c08d BootID:b414a3d0-3856-4ff4-9150-cf1f013c30b3 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:44:f9:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:44:f9:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:d9:bc:67:2c:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:55:34.434203 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.434199    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:55:34.434313 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.434271    2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:55:34.436272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.436252    2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:55:34.436409 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.436274    2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:55:34.436454 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.436411    2581 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:55:34.436454 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.436419    2581 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:55:34.436509 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.436464    2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:34.436509 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.436482    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:55:34.437285 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.437275    2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:55:34.441154 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.441141    2581 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:55:34.441280 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.441271    2581 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:55:34.442013 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.442004    2581 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:55:34.442044 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.442017    2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:55:34.442083 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.442075    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:55:34.442118 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.442087    2581 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:55:34.442118 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.442095    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:55:34.442466 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.442452    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:34.444573 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.444555    2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:55:34.451599 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.451582    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:55:34.452343 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452331    2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:55:34.452389 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452361    2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:55:34.452514 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452503    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:55:34.452550 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452524    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:55:34.452550 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452535    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:55:34.452550 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452543    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:55:34.452550 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452551    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:55:34.452672 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.452560    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:55:34.453061 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453050    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:55:34.453094 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453062    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:55:34.453094 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453070    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:55:34.453094 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453076    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:55:34.453094 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453085    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:55:34.453094 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453093    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:55:34.453858 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453848    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:55:34.453893 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.453861    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:55:34.457683 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.457669    2581 server.go:1263] "Started kubelet"
Nov 02 14:55:34.457803 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.457771    2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:55:34.457853 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.457757    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:55:34.457900 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.457854    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:55:34.458449 ip-10-0-129-45 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:55:34.458970 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.458951    2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:55:34.462590 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.462524    2581 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:55:34.466637 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.466622    2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:55:34.466732 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.466656    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:34.466798 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.466769    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:34.467029 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.467009    2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 14:55:34.467564 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467546    2581 factory.go:55] Registering systemd factory
Nov 02 14:55:34.467564 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467565    2581 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:55:34.467736 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467728    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:55:34.467791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467728    2581 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:55:34.467791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467753    2581 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:55:34.467883 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467837    2581 factory.go:153] Registering CRI-O factory
Nov 02 14:55:34.467883 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467852    2581 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:55:34.467975 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467895    2581 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:55:34.467975 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467907    2581 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:55:34.467975 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467912    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:55:34.467975 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467934    2581 factory.go:103] Registering Raw factory
Nov 02 14:55:34.467975 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.467949    2581 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:55:34.468886 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.468872    2581 manager.go:319] Starting recovery of all containers
Nov 02 14:55:34.481127 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.481108    2581 manager.go:324] Recovery completed
Nov 02 14:55:34.485261 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.485246    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:34.487166 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.487142    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:34.487246 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.487185    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:34.487246 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.487202    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:34.487741 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.487724    2581 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:55:34.487741 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.487737    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:34.487831 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.487739    2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:55:34.487831 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.487793    2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:55:34.488543 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.488528    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:34.488607 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.488547    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:34.488607 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.488561    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:34.489096 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.489075    2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 14:55:34.490036 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.490023    2581 policy_none.go:50] "Start"
Nov 02 14:55:34.490036 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.490038    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:55:34.490128 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.490049    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:55:34.490157 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.489073    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-45.ec2.internal.18e44232145b421e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-45.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-45.ec2.internal,},FirstTimestamp:2026-11-02 14:55:34.45762307 +0000 UTC m=+0.351506315,LastTimestamp:2026-11-02 14:55:34.45762307 +0000 UTC m=+0.351506315,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-45.ec2.internal,}"
Nov 02 14:55:34.491986 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.491975    2581 policy_none.go:44] "Start"
Nov 02 14:55:34.495440 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.495423    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:34.499239 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.499217    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 14:55:34.499528 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.499510    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nrfmq"
Nov 02 14:55:34.509873 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.509856    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nrfmq"
Nov 02 14:55:34.516072 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.516043    2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:55:34.518666 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.518639    2581 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:55:34.518727 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.518684    2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:55:34.518727 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.518697    2581 server.go:87] "Starting device plugin registration server"
Nov 02 14:55:34.518935 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.518923    2581 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:55:34.518974 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.518938    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:55:34.519041 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.519026    2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:55:34.519145 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.519133    2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:55:34.519145 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.519144    2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:55:34.519595 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.519577    2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:55:34.519684 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.519635    2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:34.522190 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.522121    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-45.ec2.internal.18e44232161e0cea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-45.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-45.ec2.internal,},FirstTimestamp:2026-11-02 14:55:34.487166186 +0000 UTC m=+0.381049434,LastTimestamp:2026-11-02 14:55:34.487166186 +0000 UTC m=+0.381049434,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-45.ec2.internal,}"
Nov 02 14:55:34.522297 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.522169    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 14:55:34.522297 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.522273    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 14:55:34.522641 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.522619    2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 14:55:34.591024 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.590819    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:55:34.592213 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.592196    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:55:34.592297 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.592220    2581 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:55:34.592297 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.592242    2581 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:55:34.592297 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.592279    2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 14:55:34.605535 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.605517    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:34.619057 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.619042    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:34.619815 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.619797    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:34.619901 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.619827    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:34.619901 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.619839    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:34.619901 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.619862    2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.636201 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.636155    2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.636201 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.636177    2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-45.ec2.internal\": node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:34.679351 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.679322    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:34.692635 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.692616    2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"]
Nov 02 14:55:34.692704 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.692692    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:34.693480 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.693465    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:34.693548 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.693493    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:34.693548 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.693502    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:34.695738 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.695726    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:34.695886 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.695872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.695920 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.695902    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:34.696484 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.696467    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:34.696575 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.696496    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:34.696575 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.696507    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:34.696921 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.696907    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:34.696966 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.696932    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:34.696966 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.696949    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:34.698456 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.698443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.698508 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.698468    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:34.699249 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.699234    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:34.699335 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.699269    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:34.699335 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.699283    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:34.721933 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.721916    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-45.ec2.internal\" not found" node="ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.724405 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.724378    2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-45.ec2.internal\" not found" node="ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.768900 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.768874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4d75b8d1582cef207a6bcc13a46c78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"cf4d75b8d1582cef207a6bcc13a46c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.769004 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.768899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.769004 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.768928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.779578 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.779550    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:34.869184 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.869160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.869279 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.869195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4d75b8d1582cef207a6bcc13a46c78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"cf4d75b8d1582cef207a6bcc13a46c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.869279 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.869218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.869279 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.869249    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.869279 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.869254    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4d75b8d1582cef207a6bcc13a46c78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"cf4d75b8d1582cef207a6bcc13a46c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.869279 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:34.869255    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:34.880277 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.880258    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:34.980932 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:34.980853    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.025166 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.025141    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:35.027306 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.027274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:35.081238 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.081213    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.181797 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.181766    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.282382 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.282303    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.382778 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.382753    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.401916 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.401882    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 14:55:35.402026 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.402012    2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 14:55:35.483381 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.483356    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.515097 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.515060    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:34 +0000 UTC" deadline="2028-07-05 14:16:23.047077925 +0000 UTC"
Nov 02 14:55:35.515097 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.515090    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14663h20m47.531990871s"
Nov 02 14:55:35.555881 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.555825    2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:55:35.557157 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.557137    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:55:35.557272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.557171    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:55:35.557272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.557181    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:55:35.579981 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.579963    2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:35.583922 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.583903    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.618118 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:35.618095    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4d75b8d1582cef207a6bcc13a46c78.slice/crio-091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600 WatchSource:0}: Error finding container 091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600: Status 404 returned error can't find the container with id 091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600
Nov 02 14:55:35.618824 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:35.618800    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dacd25e67dd69f0add8a7632a4c6959.slice/crio-2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570 WatchSource:0}: Error finding container 2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570: Status 404 returned error can't find the container with id 2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570
Nov 02 14:55:35.622598 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:35.622586    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:55:35.684991 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.684968    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.785414 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.785387    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.885868 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.885807    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:35.986479 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:35.986453    2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:55:36.022737 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.022714    2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:36.066816 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.066783    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:36.073900 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.073818    2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:36.090001 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.089978    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:36.091078 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.091057    2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:55:36.111204 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.111171    2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:55:36.444126 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.444099    2581 apiserver.go:52] "Watching apiserver"
Nov 02 14:55:36.468089 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.468063    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 14:55:36.485538 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.485518    2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:55:36.485878 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.485856    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-vpflc","openshift-multus/multus-additional-cni-plugins-grmcf","openshift-cluster-node-tuning-operator/tuned-kzs8g","openshift-image-registry/node-ca-7g9ww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal","openshift-network-operator/iptables-alerter-tbjb6","kube-system/global-pull-secret-syncer-l52v6","kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal","openshift-multus/multus-xw5bg","openshift-multus/network-metrics-daemon-h5wrn","openshift-network-diagnostics/network-check-target-k67qn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj","openshift-ovn-kubernetes/ovnkube-node-trwcd","kube-system/konnectivity-agent-29z4k"]
Nov 02 14:55:36.488412 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.488392    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.488517 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.488465    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:36.490386 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.490367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:36.490472 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.490431    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:36.492350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.492333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.494323 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.494306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.496794 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.496775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.499002 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.498983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.501182 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.501161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.504806 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.503034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.505239 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.505223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.507391 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.507363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:36.509913 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.509883    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:36.512701 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.512681    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.515383 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.515365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.519891 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.519874    2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 14:55:36.519984 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.519928    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8rjc"
Nov 02 14:55:36.532120 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.532099    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b8rjc"
Nov 02 14:55:36.543974 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.543951    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:55:36.544051 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.543983    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x78pf\""
Nov 02 14:55:36.544051 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.544016    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:55:36.544203 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.544185    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-btkxl\""
Nov 02 14:55:36.544272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.544227    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pvkf\""
Nov 02 14:55:36.544272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.544240    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:55:36.544272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.544245    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kws2w\""
Nov 02 14:55:36.544407 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.544294    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-62zsq\""
Nov 02 14:55:36.545323 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.545302    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7sjw\""
Nov 02 14:55:36.545423 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.545390    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-25j7m\""
Nov 02 14:55:36.545483 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.545464    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczs\""
Nov 02 14:55:36.545542 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.545518    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2fmch\""
Nov 02 14:55:36.568633 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.568610    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:55:36.578172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-k8s-cni-cncf-io\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-netns\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-modprobe-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.578272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-bin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578272 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-multus-daemon-config\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-netd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-kubelet-config\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-dbus\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-socket-dir-parent\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-run\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-etc-kubernetes\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-etc-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.578463 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-script-lib\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnz9s\" (UniqueName: \"kubernetes.io/projected/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-kube-api-access-qnz9s\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-socket-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-host\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-log-socket\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1e0c29-c679-40f0-9961-7f7ff476e04e-host\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-device-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-netns\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-lib-modules\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-var-lib-kubelet\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9q75\" (UniqueName: \"kubernetes.io/projected/3558d580-6e93-4b65-bc56-98234fa61562-kube-api-access-c9q75\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.578863 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-slash\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1e0c29-c679-40f0-9961-7f7ff476e04e-serviceca\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89t4\" (UniqueName: \"kubernetes.io/projected/7c1e0c29-c679-40f0-9961-7f7ff476e04e-kube-api-access-n89t4\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-cnibin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-conf\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.578992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-systemd\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpqm\" (UniqueName: \"kubernetes.io/projected/93eca203-b6a7-41d2-b640-953dd6ccaa02-kube-api-access-ggpqm\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-kubernetes\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10ef5542-79bd-4e48-866d-952fbeb2bad2-iptables-alerter-script\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-hostroot\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-kubelet\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-ovn\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579216    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-node-log\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-config\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-konnectivity-ca\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.579433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-systemd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-var-lib-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-registration-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-cni-binary-copy\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-tmp\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579624    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsf7\" (UniqueName: \"kubernetes.io/projected/77b573dd-a446-4605-8362-2fd3e4565d3d-kube-api-access-rtsf7\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-kubelet\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.580140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10ef5542-79bd-4e48-866d-952fbeb2bad2-host-slash\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-conf-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7dtr\" (UniqueName: \"kubernetes.io/projected/044f0c98-3b8f-4283-9bdf-d235a867e360-kube-api-access-x7dtr\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-systemd-units\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-bin\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579892    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-env-overrides\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysconfig\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-multus-certs\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.579975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhwm4\" (UniqueName: \"kubernetes.io/projected/1b56d86c-0e63-4e77-8fa7-469ebc91e866-kube-api-access-bhwm4\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-system-cni-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-sys-fs\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-system-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-tuned\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-multus\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93eca203-b6a7-41d2-b640-953dd6ccaa02-tmp-dir\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.580852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-agent-certs\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.581518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.581518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-sys\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.581518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlz4\" (UniqueName: \"kubernetes.io/projected/10ef5542-79bd-4e48-866d-952fbeb2bad2-kube-api-access-lxlz4\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.581518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovn-node-metrics-cert\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.581518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93eca203-b6a7-41d2-b640-953dd6ccaa02-hosts-file\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.581518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ppl\" (UniqueName: \"kubernetes.io/projected/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-kube-api-access-c5ppl\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:36.581518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.580325    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cnibin\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.593346 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593324    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:55:36.593555 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593533    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:55:36.593627 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593617    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:55:36.593756 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593738    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.593853 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593753    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.593853 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593775    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:55:36.593853 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593815    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:55:36.593853 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593830    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.593853 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593832    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.594143 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593856    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.594143 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593779    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.594143 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593875    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.594143 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593903    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.594143 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593936    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:55:36.594143 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.593967    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.594365 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.594349    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:55:36.594429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.594411    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.594429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.594419    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.594528 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.594425    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:55:36.594528 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.594479    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:55:36.594779 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.594760    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.597592 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.597515    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"cf4d75b8d1582cef207a6bcc13a46c78","Type":"ContainerStarted","Data":"091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600"}
Nov 02 14:55:36.598735 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.598714    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"2dacd25e67dd69f0add8a7632a4c6959","Type":"ContainerStarted","Data":"2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570"}
Nov 02 14:55:36.605807 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.605788    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:55:36.605934 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.605919    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:55:36.605997 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.605961    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:55:36.681189 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-k8s-cni-cncf-io\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-netns\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-modprobe-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-bin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-multus-daemon-config\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-netd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681273    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-netns\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-kubelet-config\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-dbus\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681308    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-bin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-socket-dir-parent\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681331    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-netd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-run\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.681360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-etc-kubernetes\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-etc-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-script-lib\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnz9s\" (UniqueName: \"kubernetes.io/projected/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-kube-api-access-qnz9s\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681423    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-modprobe-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-socket-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-host\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-log-socket\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1e0c29-c679-40f0-9961-7f7ff476e04e-host\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681517    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-run\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-device-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681560    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-kubelet-config\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681589    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-device-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-netns\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-lib-modules\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.681914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681273    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-k8s-cni-cncf-io\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-var-lib-kubelet\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9q75\" (UniqueName: \"kubernetes.io/projected/3558d580-6e93-4b65-bc56-98234fa61562-kube-api-access-c9q75\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681717    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-var-lib-kubelet\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681730    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681740    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1e0c29-c679-40f0-9961-7f7ff476e04e-host\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-slash\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681747    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-socket-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681780    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-socket-dir-parent\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1e0c29-c679-40f0-9961-7f7ff476e04e-serviceca\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681821    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-log-socket\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681828    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-etc-kubernetes\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89t4\" (UniqueName: \"kubernetes.io/projected/7c1e0c29-c679-40f0-9961-7f7ff476e04e-kube-api-access-n89t4\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681862    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-netns\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681875    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681887    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-cnibin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681930    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-cnibin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681961    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-etc-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.681989    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-lib-modules\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682018    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-conf\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682023    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-dbus\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-systemd\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-host\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpqm\" (UniqueName: \"kubernetes.io/projected/93eca203-b6a7-41d2-b640-953dd6ccaa02-kube-api-access-ggpqm\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682114    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-systemd\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-kubernetes\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10ef5542-79bd-4e48-866d-952fbeb2bad2-iptables-alerter-script\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-hostroot\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-kubelet\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-ovn\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682225    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-conf\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-node-log\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682254    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-config\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-konnectivity-ca\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682295    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-hostroot\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.682992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1e0c29-c679-40f0-9961-7f7ff476e04e-serviceca\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682363    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-node-log\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-systemd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682407    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-kubernetes\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-var-lib-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682441    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-ovn\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682477    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-slash\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-registration-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682560    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-registration-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-systemd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.683620 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-cni-binary-copy\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-tmp\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsf7\" (UniqueName: \"kubernetes.io/projected/77b573dd-a446-4605-8362-2fd3e4565d3d-kube-api-access-rtsf7\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-kubelet\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682812    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-config\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10ef5542-79bd-4e48-866d-952fbeb2bad2-host-slash\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682813    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-konnectivity-ca\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682849    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-kubelet\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-conf-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682483    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-script-lib\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682892    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-conf-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682886    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7dtr\" (UniqueName: \"kubernetes.io/projected/044f0c98-3b8f-4283-9bdf-d235a867e360-kube-api-access-x7dtr\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-systemd-units\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.684350 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.682970    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-bin\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682986    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-env-overrides\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.683027    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs podName:4f1313c9-5c99-44a9-914f-3bdbcfd9a23f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.183010255 +0000 UTC m=+3.076893510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs") pod "network-metrics-daemon-h5wrn" (UID: "4f1313c9-5c99-44a9-914f-3bdbcfd9a23f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysconfig\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-multus-certs\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683102    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwm4\" (UniqueName: \"kubernetes.io/projected/1b56d86c-0e63-4e77-8fa7-469ebc91e866-kube-api-access-bhwm4\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-system-cni-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.683166    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-sys-fs\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683173    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10ef5542-79bd-4e48-866d-952fbeb2bad2-iptables-alerter-script\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-system-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.683198    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret podName:35254468-9d2e-4c69-a162-02ae0854b07f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.183189246 +0000 UTC m=+3.077072476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret") pod "global-pull-secret-syncer-l52v6" (UID: "35254468-9d2e-4c69-a162-02ae0854b07f") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683224    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-multus-certs\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.685063 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683230    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683258    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.682667    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-var-lib-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683265    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-system-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683280    2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-tuned\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683314    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-env-overrides\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-multus\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683354    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93eca203-b6a7-41d2-b640-953dd6ccaa02-tmp-dir\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683386    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-system-cni-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-agent-certs\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-sys\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683456    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-multus\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlz4\" (UniqueName: \"kubernetes.io/projected/10ef5542-79bd-4e48-866d-952fbeb2bad2-kube-api-access-lxlz4\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683479    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10ef5542-79bd-4e48-866d-952fbeb2bad2-host-slash\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683498    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysconfig\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.685791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovn-node-metrics-cert\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683523    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-systemd-units\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683523    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-kubelet\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93eca203-b6a7-41d2-b640-953dd6ccaa02-hosts-file\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683577    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93eca203-b6a7-41d2-b640-953dd6ccaa02-hosts-file\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ppl\" (UniqueName: \"kubernetes.io/projected/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-kube-api-access-c5ppl\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cnibin\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cnibin\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683792    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-cni-binary-copy\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-sys-fs\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683792    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-bin\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.683989    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.684026    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93eca203-b6a7-41d2-b640-953dd6ccaa02-tmp-dir\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.684208    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-sys\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.684211    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.686531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.684798    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-multus-daemon-config\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.687206 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.687102    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-tuned\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.687206 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.687119    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-tmp\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.687415 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.687393    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-agent-certs\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.688028 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.688007    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovn-node-metrics-cert\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.703894 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.703806    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:36.703894 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.703838    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:36.703894 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.703852    2581 projected.go:196] Error preparing data for projected volume kube-api-access-65ps9 for pod openshift-network-diagnostics/network-check-target-k67qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:36.704190 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:36.703914    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9 podName:ce469d24-f9cc-4381-be8a-5c9006c21db7 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:37.203895446 +0000 UTC m=+3.097778695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-65ps9" (UniqueName: "kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9") pod "network-check-target-k67qn" (UID: "ce469d24-f9cc-4381-be8a-5c9006c21db7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:36.704780 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.704759    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpqm\" (UniqueName: \"kubernetes.io/projected/93eca203-b6a7-41d2-b640-953dd6ccaa02-kube-api-access-ggpqm\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.704989 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.704970    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9q75\" (UniqueName: \"kubernetes.io/projected/3558d580-6e93-4b65-bc56-98234fa61562-kube-api-access-c9q75\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.705289 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.705271    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89t4\" (UniqueName: \"kubernetes.io/projected/7c1e0c29-c679-40f0-9961-7f7ff476e04e-kube-api-access-n89t4\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.705432 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.705413    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnz9s\" (UniqueName: \"kubernetes.io/projected/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-kube-api-access-qnz9s\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.708782 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.708761    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsf7\" (UniqueName: \"kubernetes.io/projected/77b573dd-a446-4605-8362-2fd3e4565d3d-kube-api-access-rtsf7\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.710103 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.710052    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7dtr\" (UniqueName: \"kubernetes.io/projected/044f0c98-3b8f-4283-9bdf-d235a867e360-kube-api-access-x7dtr\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.710103 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.710066    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ppl\" (UniqueName: \"kubernetes.io/projected/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-kube-api-access-c5ppl\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:36.710243 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.710122    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlz4\" (UniqueName: \"kubernetes.io/projected/10ef5542-79bd-4e48-866d-952fbeb2bad2-kube-api-access-lxlz4\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.711341 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.711323    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwm4\" (UniqueName: \"kubernetes.io/projected/1b56d86c-0e63-4e77-8fa7-469ebc91e866-kube-api-access-bhwm4\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.806973 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.806947    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:55:36.814181 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.814158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:55:36.815597 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:36.815570    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b573dd_a446_4605_8362_2fd3e4565d3d.slice/crio-3479b33f169f627dbaca657c8b826ea1e09d2798c233a5ecc25bcdd1c127af85 WatchSource:0}: Error finding container 3479b33f169f627dbaca657c8b826ea1e09d2798c233a5ecc25bcdd1c127af85: Status 404 returned error can't find the container with id 3479b33f169f627dbaca657c8b826ea1e09d2798c233a5ecc25bcdd1c127af85
Nov 02 14:55:36.823617 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.823594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:36.829938 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.829922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:55:36.837210 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.837195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:55:36.843157 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.843142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xw5bg"
Nov 02 14:55:36.852164 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:36.852139    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ef5542_79bd_4e48_866d_952fbeb2bad2.slice/crio-94f7ce2e142253298f792915915ec82a3c36e6019171a57aa3672b45879a73fc WatchSource:0}: Error finding container 94f7ce2e142253298f792915915ec82a3c36e6019171a57aa3672b45879a73fc: Status 404 returned error can't find the container with id 94f7ce2e142253298f792915915ec82a3c36e6019171a57aa3672b45879a73fc
Nov 02 14:55:36.853091 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:36.853057    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab14eae_d5f0_46e3_abda_c817d0e8b66e.slice/crio-937c3df4f33852f5d87707288b369b5eaf7f03fbff9c0993c12444275c8fbc18 WatchSource:0}: Error finding container 937c3df4f33852f5d87707288b369b5eaf7f03fbff9c0993c12444275c8fbc18: Status 404 returned error can't find the container with id 937c3df4f33852f5d87707288b369b5eaf7f03fbff9c0993c12444275c8fbc18
Nov 02 14:55:36.873687 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.873632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:36.877419 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.877396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:55:36.883518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.883498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:55:36.891272 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:36.891237    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044f0c98_3b8f_4283_9bdf_d235a867e360.slice/crio-f1265bf239a70818a94c4f7b7196faed9734500524f5a4bb22c5549b845b5fbe WatchSource:0}: Error finding container f1265bf239a70818a94c4f7b7196faed9734500524f5a4bb22c5549b845b5fbe: Status 404 returned error can't find the container with id f1265bf239a70818a94c4f7b7196faed9734500524f5a4bb22c5549b845b5fbe
Nov 02 14:55:36.894528 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:55:36.894508    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7d86c9_eec3_46af_91eb_e1a2851a59d4.slice/crio-3fcd9979316bc5a76dd6faa25d7dfc0568e064c067ec99385d312b46b745e04f WatchSource:0}: Error finding container 3fcd9979316bc5a76dd6faa25d7dfc0568e064c067ec99385d312b46b745e04f: Status 404 returned error can't find the container with id 3fcd9979316bc5a76dd6faa25d7dfc0568e064c067ec99385d312b46b745e04f
Nov 02 14:55:36.999842 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:36.999777    2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:55:37.186000 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.185833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:37.186133 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.186024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:37.186133 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.186071    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:37.186133 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.186112    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:37.186133 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.186123    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret podName:35254468-9d2e-4c69-a162-02ae0854b07f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:38.186109887 +0000 UTC m=+4.079993131 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret") pod "global-pull-secret-syncer-l52v6" (UID: "35254468-9d2e-4c69-a162-02ae0854b07f") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:37.186281 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.186155    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs podName:4f1313c9-5c99-44a9-914f-3bdbcfd9a23f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:38.186144574 +0000 UTC m=+4.080027822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs") pod "network-metrics-daemon-h5wrn" (UID: "4f1313c9-5c99-44a9-914f-3bdbcfd9a23f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:37.286830 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.286721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:37.286974 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.286937    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:37.286974 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.286957    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:37.286974 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.286968    2581 projected.go:196] Error preparing data for projected volume kube-api-access-65ps9 for pod openshift-network-diagnostics/network-check-target-k67qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:37.287119 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:37.287025    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9 podName:ce469d24-f9cc-4381-be8a-5c9006c21db7 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:38.287004434 +0000 UTC m=+4.180887685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-65ps9" (UniqueName: "kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9") pod "network-check-target-k67qn" (UID: "ce469d24-f9cc-4381-be8a-5c9006c21db7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:37.535532 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.535435    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:36 +0000 UTC" deadline="2028-06-11 02:14:35.723616213 +0000 UTC"
Nov 02 14:55:37.535532 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.535485    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14075h18m58.188135027s"
Nov 02 14:55:37.617699 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.617313    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerStarted","Data":"3fcd9979316bc5a76dd6faa25d7dfc0568e064c067ec99385d312b46b745e04f"}
Nov 02 14:55:37.625329 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.625302    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj" event={"ID":"044f0c98-3b8f-4283-9bdf-d235a867e360","Type":"ContainerStarted","Data":"f1265bf239a70818a94c4f7b7196faed9734500524f5a4bb22c5549b845b5fbe"}
Nov 02 14:55:37.628927 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.628903    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"c685a0bde3520f72756305ed36f255dd468021b39de0470f942090a124b62bdf"}
Nov 02 14:55:37.638500 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.638408    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xw5bg" event={"ID":"3558d580-6e93-4b65-bc56-98234fa61562","Type":"ContainerStarted","Data":"ecaaaf25edd0c569d11c4cc78a243e0d83beb332dca49c00d5f1fe35c5d8ff21"}
Nov 02 14:55:37.651752 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.651727    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-29z4k" event={"ID":"4ab14eae-d5f0-46e3-abda-c817d0e8b66e","Type":"ContainerStarted","Data":"937c3df4f33852f5d87707288b369b5eaf7f03fbff9c0993c12444275c8fbc18"}
Nov 02 14:55:37.667482 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.667459    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tbjb6" event={"ID":"10ef5542-79bd-4e48-866d-952fbeb2bad2","Type":"ContainerStarted","Data":"94f7ce2e142253298f792915915ec82a3c36e6019171a57aa3672b45879a73fc"}
Nov 02 14:55:37.670906 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.670881    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7g9ww" event={"ID":"7c1e0c29-c679-40f0-9961-7f7ff476e04e","Type":"ContainerStarted","Data":"49e27abbc9f345773c85b7f2ddcd2adb929d522ffbd976558bfcc2fd0d7de287"}
Nov 02 14:55:37.686079 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.686020    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"cf4d75b8d1582cef207a6bcc13a46c78","Type":"ContainerStarted","Data":"56bfbd8971a55248aa6511ebb93f2c8a88c50a4013813b14238f7ccc0ff6805a"}
Nov 02 14:55:37.694424 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.694354    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vpflc" event={"ID":"93eca203-b6a7-41d2-b640-953dd6ccaa02","Type":"ContainerStarted","Data":"fe7861a94ec7e5837a6b3685d9dda34fb14c06f334b490e720ce085022c6e98a"}
Nov 02 14:55:37.699019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.698997    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzs8g" event={"ID":"77b573dd-a446-4605-8362-2fd3e4565d3d","Type":"ContainerStarted","Data":"3479b33f169f627dbaca657c8b826ea1e09d2798c233a5ecc25bcdd1c127af85"}
Nov 02 14:55:37.715478 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.714902    2581 generic.go:356] "Generic (PLEG): container finished" podID="2dacd25e67dd69f0add8a7632a4c6959" containerID="06baa112e4e66e7e61137707c5e44136be80145eb042faa388eab187c44556ad" exitCode=0
Nov 02 14:55:37.715478 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.714939    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"2dacd25e67dd69f0add8a7632a4c6959","Type":"ContainerDied","Data":"06baa112e4e66e7e61137707c5e44136be80145eb042faa388eab187c44556ad"}
Nov 02 14:55:37.764761 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:37.764331    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" podStartSLOduration=2.764312893 podStartE2EDuration="2.764312893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:37.725679239 +0000 UTC m=+3.619562494" watchObservedRunningTime="2026-11-02 14:55:37.764312893 +0000 UTC m=+3.658196145"
Nov 02 14:55:38.206840 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.206180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:38.206840 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.206261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:38.206840 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.206268    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:38.206840 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.206337    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:38.206840 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.206382    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret podName:35254468-9d2e-4c69-a162-02ae0854b07f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:40.206362733 +0000 UTC m=+6.100245977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret") pod "global-pull-secret-syncer-l52v6" (UID: "35254468-9d2e-4c69-a162-02ae0854b07f") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:38.206840 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.206408    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs podName:4f1313c9-5c99-44a9-914f-3bdbcfd9a23f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:40.206390761 +0000 UTC m=+6.100273996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs") pod "network-metrics-daemon-h5wrn" (UID: "4f1313c9-5c99-44a9-914f-3bdbcfd9a23f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:38.307729 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.307160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:38.307729 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.307349    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:38.307729 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.307370    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:38.307729 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.307380    2581 projected.go:196] Error preparing data for projected volume kube-api-access-65ps9 for pod openshift-network-diagnostics/network-check-target-k67qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:38.307729 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.307422    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9 podName:ce469d24-f9cc-4381-be8a-5c9006c21db7 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:40.307410872 +0000 UTC m=+6.201294102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-65ps9" (UniqueName: "kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9") pod "network-check-target-k67qn" (UID: "ce469d24-f9cc-4381-be8a-5c9006c21db7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:38.536226 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.536088    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:36 +0000 UTC" deadline="2028-05-08 20:30:20.255918642 +0000 UTC"
Nov 02 14:55:38.536226 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.536155    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13277h34m41.719768258s"
Nov 02 14:55:38.595300 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.594836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:38.595300 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.594893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:38.595300 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.594893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:38.595300 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.594986    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:38.595300 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.595066    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:38.595300 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:38.595159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:38.731094 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.731051    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"2dacd25e67dd69f0add8a7632a4c6959","Type":"ContainerStarted","Data":"31a45f2899ce79bac5a6683e3a8142d7d7d6a77896326af651c7f96737b91f97"}
Nov 02 14:55:38.772081 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:38.771829    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" podStartSLOduration=3.771811719 podStartE2EDuration="3.771811719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:38.770705757 +0000 UTC m=+4.664589009" watchObservedRunningTime="2026-11-02 14:55:38.771811719 +0000 UTC m=+4.665694971"
Nov 02 14:55:39.736973 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:39.736889    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="ce08091ec56539fcbc9edef4581568bb63a859095b81a56c1bdf4e7b12e10d56" exitCode=0
Nov 02 14:55:39.737963 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:39.737936    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerDied","Data":"ce08091ec56539fcbc9edef4581568bb63a859095b81a56c1bdf4e7b12e10d56"}
Nov 02 14:55:40.222864 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.222021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:40.222864 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.222075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:40.222864 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.222160    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:40.222864 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.222187    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:40.222864 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.222258    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret podName:35254468-9d2e-4c69-a162-02ae0854b07f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.222226735 +0000 UTC m=+10.116109970 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret") pod "global-pull-secret-syncer-l52v6" (UID: "35254468-9d2e-4c69-a162-02ae0854b07f") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:40.222864 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.222282    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs podName:4f1313c9-5c99-44a9-914f-3bdbcfd9a23f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.222269729 +0000 UTC m=+10.116152964 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs") pod "network-metrics-daemon-h5wrn" (UID: "4f1313c9-5c99-44a9-914f-3bdbcfd9a23f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:40.322528 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.322492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:40.322777 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.322748    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:40.322841 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.322785    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:40.322841 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.322799    2581 projected.go:196] Error preparing data for projected volume kube-api-access-65ps9 for pod openshift-network-diagnostics/network-check-target-k67qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:40.322960 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.322859    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9 podName:ce469d24-f9cc-4381-be8a-5c9006c21db7 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:44.322840542 +0000 UTC m=+10.216723775 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-65ps9" (UniqueName: "kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9") pod "network-check-target-k67qn" (UID: "ce469d24-f9cc-4381-be8a-5c9006c21db7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:40.594051 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.593525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:40.594051 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.593660    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:40.594051 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.593720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:40.594051 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.593801    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:40.594051 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.593873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:40.594051 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:40.593966    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:40.746702 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.745933    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-29z4k" event={"ID":"4ab14eae-d5f0-46e3-abda-c817d0e8b66e","Type":"ContainerStarted","Data":"706d6db71a9a3e4d417a19a56a0315c6d435ae243c86355860328c43f9d53707"}
Nov 02 14:55:40.790499 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:40.790439    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-29z4k" podStartSLOduration=3.400107364 podStartE2EDuration="6.79042307s" totalImagesPullingTime="3.390315706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:40.789997136 +0000 UTC m=+6.683880391" watchObservedRunningTime="2026-11-02 14:55:40.79042307 +0000 UTC m=+6.684306326"
Nov 02 14:55:41.760800 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:41.760766    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7g9ww" event={"ID":"7c1e0c29-c679-40f0-9961-7f7ff476e04e","Type":"ContainerStarted","Data":"9515e1d18586cc20bb8110ef237b7772fb8c1458a5006f3c0903087a3e2896b5"}
Nov 02 14:55:42.592895 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:42.592820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:42.593056 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:42.592937    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:42.593285 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:42.593266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:42.593399 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:42.593378    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:42.593483 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:42.593460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:42.593560 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:42.593542    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:42.764783 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:42.764701    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj" event={"ID":"044f0c98-3b8f-4283-9bdf-d235a867e360","Type":"ContainerStarted","Data":"3015002573c0cf332aedd6b7b1868ded56b2b4be9175d0b74da2096fd3fac743"}
Nov 02 14:55:42.797614 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:42.797495    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:42.936215 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:42.936182    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:43.009858 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.009246    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:43.112855 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.112765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:43.407779 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.407707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:43.552251 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.552121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:43.709848 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.709711    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:43.710255 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.710186    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:43.711080 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.710897    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:43.754026 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.753981    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7g9ww" podStartSLOduration=5.243713134 podStartE2EDuration="9.753965255s" totalImagesPullingTime="4.510252121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:41.847803301 +0000 UTC m=+7.741686555" watchObservedRunningTime="2026-11-02 14:55:43.753965255 +0000 UTC m=+9.647848509"
Nov 02 14:55:43.767342 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.767196    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:55:43.847269 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.847100    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:43.921056 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.921034    2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:55:43.991708 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:43.991683    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:44.268753 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.268721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:44.268869 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.268772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:44.268933 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.268904    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:44.268989 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.268973    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret podName:35254468-9d2e-4c69-a162-02ae0854b07f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.268953012 +0000 UTC m=+18.162836260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret") pod "global-pull-secret-syncer-l52v6" (UID: "35254468-9d2e-4c69-a162-02ae0854b07f") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:44.269289 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.268906    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:44.269384 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.269317    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs podName:4f1313c9-5c99-44a9-914f-3bdbcfd9a23f nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.269298482 +0000 UTC m=+18.163181719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs") pod "network-metrics-daemon-h5wrn" (UID: "4f1313c9-5c99-44a9-914f-3bdbcfd9a23f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:44.369404 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.369371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:44.369584 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.369562    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:44.369678 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.369587    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:44.369678 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.369600    2581 projected.go:196] Error preparing data for projected volume kube-api-access-65ps9 for pod openshift-network-diagnostics/network-check-target-k67qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:44.369678 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.369673    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9 podName:ce469d24-f9cc-4381-be8a-5c9006c21db7 nodeName:}" failed. No retries permitted until 2026-11-02 14:55:52.369634163 +0000 UTC m=+18.263517409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-65ps9" (UniqueName: "kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9") pod "network-check-target-k67qn" (UID: "ce469d24-f9cc-4381-be8a-5c9006c21db7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:44.523383 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.523121    2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:55:43.921053174Z","UUID":"7094a2e6-bae2-489a-9d5e-a2fd5ea6857f","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:55:44.532115 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.531849    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:55:44.532115 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.531879    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:55:44.542900 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.542868    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:44.594030 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.594004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:44.594178 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.594029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:44.594236 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.594191    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:44.594298 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.594211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:44.594349 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.594294    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:44.594349 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:44.594333    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:44.693918 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.693879    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:44.771154 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.771082    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="5d959f778ba0ca8bb477b0fa3285b46377816892e7d21eea2f0b50f44362d566" exitCode=0
Nov 02 14:55:44.771572 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.771161    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerDied","Data":"5d959f778ba0ca8bb477b0fa3285b46377816892e7d21eea2f0b50f44362d566"}
Nov 02 14:55:44.774684 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.774659    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj" event={"ID":"044f0c98-3b8f-4283-9bdf-d235a867e360","Type":"ContainerStarted","Data":"94cd359440144a65f036979fabfa68406ae0460ea019e57b3e07bb7b3452186b"}
Nov 02 14:55:44.776226 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.776174    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xw5bg" event={"ID":"3558d580-6e93-4b65-bc56-98234fa61562","Type":"ContainerStarted","Data":"5cf53c8e189fa33dd72726f5755ca2dba7d0b1df248b40044366bc1e5fbe7f5e"}
Nov 02 14:55:44.778916 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.778895    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vpflc" event={"ID":"93eca203-b6a7-41d2-b640-953dd6ccaa02","Type":"ContainerStarted","Data":"2e21e7f20d6a15e7a5145ec7b2a1177efe5163f1c6ec3196ee5188817777fe63"}
Nov 02 14:55:44.908740 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:44.908690    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vpflc" podStartSLOduration=3.591373973 podStartE2EDuration="10.908671109s" totalImagesPullingTime="7.317297136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:44.863271072 +0000 UTC m=+10.757154328" watchObservedRunningTime="2026-11-02 14:55:44.908671109 +0000 UTC m=+10.802554363"
Nov 02 14:55:45.768994 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:45.768962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:45.783801 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:45.783213    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="f14323aad401518ec466327867954446d8af199699960d76ceeb7b00c3b4af82" exitCode=0
Nov 02 14:55:45.783801 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:45.783316    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerDied","Data":"f14323aad401518ec466327867954446d8af199699960d76ceeb7b00c3b4af82"}
Nov 02 14:55:45.787099 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:45.787078    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj" event={"ID":"044f0c98-3b8f-4283-9bdf-d235a867e360","Type":"ContainerStarted","Data":"5fb0fdd271e042404f1a04131611f5e488ce8c443957303f0c9533a382d3b911"}
Nov 02 14:55:45.893387 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:45.893339    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xw5bg" podStartSLOduration=4.809718479 podStartE2EDuration="11.893319813s" totalImagesPullingTime="7.083601334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:44.909381716 +0000 UTC m=+10.803264970" watchObservedRunningTime="2026-11-02 14:55:45.893319813 +0000 UTC m=+11.787203067"
Nov 02 14:55:45.930764 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:45.930681    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:46.383403 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.383362    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:46.550355 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.550322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:46.593638 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.593561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:46.593638 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.593591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:46.593868 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:46.593725    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:46.593868 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:46.593802    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:46.593868 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.593837    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:46.594018 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:46.593884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:46.750888 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.750863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:46.791178 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.791145    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tbjb6" event={"ID":"10ef5542-79bd-4e48-866d-952fbeb2bad2","Type":"ContainerStarted","Data":"d070ed55c59b268f0390791869410b2dc6e95f949f39f07048ff10ba10a53e0d"}
Nov 02 14:55:46.812041 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.812020    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vpflc_93eca203-b6a7-41d2-b640-953dd6ccaa02/dns-node-resolver/0.log"
Nov 02 14:55:46.836387 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.836323    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj" podStartSLOduration=5.281142555 podStartE2EDuration="12.836304822s" totalImagesPullingTime="7.555162267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:45.938392796 +0000 UTC m=+11.832276052" watchObservedRunningTime="2026-11-02 14:55:46.836304822 +0000 UTC m=+12.730188077"
Nov 02 14:55:46.904923 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:46.904876    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:47.215693 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.215659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:47.394836 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.394805    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:47.412229 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.412201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7g9ww_7c1e0c29-c679-40f0-9961-7f7ff476e04e/node-ca/0.log"
Nov 02 14:55:47.604224 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.604138    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:47.795013 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.794976    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzs8g" event={"ID":"77b573dd-a446-4605-8362-2fd3e4565d3d","Type":"ContainerStarted","Data":"ca4cc51e5b2b126d914d83e2e2e8080354c5eb2e11bad4575867690b37fb8079"}
Nov 02 14:55:47.797746 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.797723    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="2837e0eff8b3a7178c643708d5c8316506232522d49aefab2f004b84a99a5996" exitCode=0
Nov 02 14:55:47.797847 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.797811    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerDied","Data":"2837e0eff8b3a7178c643708d5c8316506232522d49aefab2f004b84a99a5996"}
Nov 02 14:55:47.840008 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.839956    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kzs8g" podStartSLOduration=3.911891974 podStartE2EDuration="13.83994234s" totalImagesPullingTime="9.928050366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:47.839799186 +0000 UTC m=+13.733682441" watchObservedRunningTime="2026-11-02 14:55:47.83994234 +0000 UTC m=+13.733825593"
Nov 02 14:55:47.840178 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.840151    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tbjb6" podStartSLOduration=6.154735756 podStartE2EDuration="13.840143761s" totalImagesPullingTime="7.685408005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:46.846659722 +0000 UTC m=+12.740542973" watchObservedRunningTime="2026-11-02 14:55:47.840143761 +0000 UTC m=+13.734027017"
Nov 02 14:55:47.999488 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:47.999454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:48.592817 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:48.592679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:48.592817 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:48.592766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:48.593036 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:48.592882    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:48.593036 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:48.592891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:48.593036 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:48.592942    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:48.593036 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:48.593024    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:48.593237 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:48.593087    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:49.198999 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:49.198767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:49.404661 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:49.404596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:49.794076 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:49.793991    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:49.805257 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:49.805227    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="1c212160d005aaf5e1653879fe3e829e4eb291c840452dda1933c1d7647b9419" exitCode=0
Nov 02 14:55:49.805401 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:49.805290    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerDied","Data":"1c212160d005aaf5e1653879fe3e829e4eb291c840452dda1933c1d7647b9419"}
Nov 02 14:55:49.925918 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:49.925884    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:50.068042 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.067961    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:50.206455 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.206382    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:50.593073 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.593045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:50.593244 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.593134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:50.593304 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:50.593247    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:50.593304 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:50.593137    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:50.593386 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.593304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:50.593433 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:50.593410    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:50.602156 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.602127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:50.754997 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.754962    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:50.809932 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.809903    2581 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="6a7d158955cc8706d1ca90bf9e20f0677a56dd6cc3b840f38c19268d4f63719d" exitCode=0
Nov 02 14:55:50.810064 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:50.809952    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerDied","Data":"6a7d158955cc8706d1ca90bf9e20f0677a56dd6cc3b840f38c19268d4f63719d"}
Nov 02 14:55:51.206771 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.206592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:51.400044 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.399973    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:51.798445 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.798365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:51.814818 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.814787    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grmcf" event={"ID":"8e7d86c9-eec3-46af-91eb-e1a2851a59d4","Type":"ContainerStarted","Data":"ed85bb6bc40b9a3ea31c2c9a77835105ef15acb9ce4e0b5e4eb0627c12dc23bd"}
Nov 02 14:55:51.817461 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.817442    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"1584d745b61641d4617e223339eae378bfc0fdbb67e5d52fa065acd85ac67ab4"}
Nov 02 14:55:51.817537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.817502    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"21563edf4d1eb2dbec52bc29f5764b5a217fb82bc2e8af713246478900a0a78d"}
Nov 02 14:55:51.817537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.817535    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"9d7691598be248d30e4a8b10697292806d7e89bbec8254b849c0d8ffb090fb8e"}
Nov 02 14:55:51.817605 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.817562    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"5310f50ed3ab36eeea4bab431239ae2dff42263566f70204b995ae4743eec999"}
Nov 02 14:55:51.817605 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.817588    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"0841659a1298f5999f00bbeef7c0bd94b844fc2585df13b5c7b45a8331d3924d"}
Nov 02 14:55:51.817669 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.817616    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"2fbba8bcc26e807e4a85b3e0d2a9c443244f404d6cba2e5d5637d912ad64453a"}
Nov 02 14:55:51.866427 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:51.866386    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-grmcf" podStartSLOduration=7.899752745 podStartE2EDuration="17.866373855s" totalImagesPullingTime="9.96662111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:51.8641495 +0000 UTC m=+17.758032754" watchObservedRunningTime="2026-11-02 14:55:51.866373855 +0000 UTC m=+17.760257108"
Nov 02 14:55:52.013293 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.013261    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:52.328757 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.328727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:52.328757 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.328763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:52.329211 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.328868    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:52.329211 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.328921    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret podName:35254468-9d2e-4c69-a162-02ae0854b07f nodeName:}" failed. No retries permitted until 2026-11-02 14:56:08.32890706 +0000 UTC m=+34.222790295 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret") pod "global-pull-secret-syncer-l52v6" (UID: "35254468-9d2e-4c69-a162-02ae0854b07f") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:55:52.329211 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.328868    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:52.329211 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.328984    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs podName:4f1313c9-5c99-44a9-914f-3bdbcfd9a23f nodeName:}" failed. No retries permitted until 2026-11-02 14:56:08.328973443 +0000 UTC m=+34.222856692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs") pod "network-metrics-daemon-h5wrn" (UID: "4f1313c9-5c99-44a9-914f-3bdbcfd9a23f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:55:52.429344 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.429316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:52.429498 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.429433    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:55:52.429498 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.429446    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:55:52.429498 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.429455    2581 projected.go:196] Error preparing data for projected volume kube-api-access-65ps9 for pod openshift-network-diagnostics/network-check-target-k67qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:52.429498 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.429497    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9 podName:ce469d24-f9cc-4381-be8a-5c9006c21db7 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:08.429485532 +0000 UTC m=+34.323368764 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-65ps9" (UniqueName: "kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9") pod "network-check-target-k67qn" (UID: "ce469d24-f9cc-4381-be8a-5c9006c21db7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:55:52.593464 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.593388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:52.593464 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.593433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:52.593666 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.593538    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:52.593724 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.593663    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:52.593724 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.593700    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:52.593833 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:52.593812    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:52.594504 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.594486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:52.752892 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:52.752863    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:53.266357 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:53.266322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:53.412939 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:53.412905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:53.633631 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:53.633600    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:53.788352 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:53.788315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:53.824518 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:53.824490    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"c2018b157d2ba2f2b76c7e3715eaeb7486d94b53656207ead1b09ee9ba9993a0"}
Nov 02 14:55:53.995539 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:53.995450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:54.594298 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:54.594270    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:54.594833 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:54.594354    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:54.594833 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:54.594377    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:54.594833 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:54.594437    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:54.594833 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:54.594447    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:54.594833 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:54.594593    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:54.595182 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:54.595164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:54.877824 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:54.877792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:55.192207 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.192121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:55.393957 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.393925    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:55.593490 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.593456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:55.792715 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.792598    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:55.832140 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.832111    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" event={"ID":"1b56d86c-0e63-4e77-8fa7-469ebc91e866","Type":"ContainerStarted","Data":"8f247695f7151370a27a08e2be9e9146f0c52ab70369b5bd9889d74367a18487"}
Nov 02 14:55:55.832414 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.832388    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:55.832522 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.832449    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:55.847592 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.847571    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:55.934585 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:55.934532    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd" podStartSLOduration=7.844015105 podStartE2EDuration="21.934518935s" totalImagesPullingTime="14.09050383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:55.918781333 +0000 UTC m=+21.812664587" watchObservedRunningTime="2026-11-02 14:55:55.934518935 +0000 UTC m=+21.828402188"
Nov 02 14:55:56.593445 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:56.593414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:56.593445 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:56.593432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:56.593668 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:56.593491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:56.593668 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:56.593584    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:56.593668 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:56.593634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:56.593814 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:56.593730    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:56.604409 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:56.604383    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:56.748084 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:56.748050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:56.834483 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:56.834451    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:56.848949 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:56.848892    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:55:58.592792 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:58.592756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:55:58.592792 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:58.592786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:55:58.593441 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:58.592868    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:55:58.593441 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:58.592909    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:55:58.593441 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:58.592953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:55:58.593441 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:55:58.593029    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:55:58.884276 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:58.884237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:59.501385 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:59.501352    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:59.644822 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:59.644779    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:55:59.873942 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:55:59.873914    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:00.016773 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.016743    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:00.532449 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.532418    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:00.592659 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.592626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:00.592805 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.592626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:00.592805 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:00.592748    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:56:00.592917 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:00.592824    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:56:00.592917 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.592632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:00.592917 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:00.592894    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:56:00.685205 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.685175    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:00.823235 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.823157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:00.957259 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:00.957228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:01.097055 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:01.096975    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:01.243759 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:01.243728    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:02.592640 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:02.592614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:02.593029 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:02.592618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:02.593029 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:02.592732    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:56:02.593029 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:02.592621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:02.593029 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:02.592787    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:56:02.593029 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:02.592889    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:56:02.718329 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:02.718301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:02.867321 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:02.867290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:04.593361 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:04.593333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:04.593778 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:04.593420    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:56:04.593778 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:04.593501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:04.593778 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:04.593600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:04.593778 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:04.593625    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:56:04.593778 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:04.593692    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:56:05.643911 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:05.643877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:05.790795 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:05.790758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:06.592809 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:06.592774    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:06.592809 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:06.592795    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:06.593070 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:06.592788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:06.593070 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:06.592880    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:56:06.593070 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:06.592975    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:56:06.593070 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:06.593027    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:56:07.355446 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:07.355406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:08.343282 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.343251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:08.343410 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.343289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:08.343410 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.343385    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:56:08.343410 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.343398    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 14:56:08.343549 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.343436    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs podName:4f1313c9-5c99-44a9-914f-3bdbcfd9a23f nodeName:}" failed. No retries permitted until 2026-11-02 14:56:40.343422506 +0000 UTC m=+66.237305751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs") pod "network-metrics-daemon-h5wrn" (UID: "4f1313c9-5c99-44a9-914f-3bdbcfd9a23f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 14:56:08.343549 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.343460    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret podName:35254468-9d2e-4c69-a162-02ae0854b07f nodeName:}" failed. No retries permitted until 2026-11-02 14:56:40.343441163 +0000 UTC m=+66.237324409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret") pod "global-pull-secret-syncer-l52v6" (UID: "35254468-9d2e-4c69-a162-02ae0854b07f") : object "kube-system"/"original-pull-secret" not registered
Nov 02 14:56:08.444110 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.444080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:08.444487 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.444199    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 14:56:08.444487 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.444213    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 14:56:08.444487 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.444221    2581 projected.go:196] Error preparing data for projected volume kube-api-access-65ps9 for pod openshift-network-diagnostics/network-check-target-k67qn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:56:08.444487 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.444273    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9 podName:ce469d24-f9cc-4381-be8a-5c9006c21db7 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:40.444259112 +0000 UTC m=+66.338142347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-65ps9" (UniqueName: "kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9") pod "network-check-target-k67qn" (UID: "ce469d24-f9cc-4381-be8a-5c9006c21db7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 14:56:08.568009 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.567976    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k67qn"]
Nov 02 14:56:08.568194 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.568177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:08.568319 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.568298    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:56:08.571204 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.571171    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l52v6"]
Nov 02 14:56:08.571339 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.571300    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:08.571421 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.571399    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:56:08.571823 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.571798    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5wrn"]
Nov 02 14:56:08.571918 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:08.571902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:08.572022 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:08.572002    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:56:09.463987 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:09.463943    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:09.645091 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:09.644924    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vpflc_93eca203-b6a7-41d2-b640-953dd6ccaa02/dns-node-resolver/0.log"
Nov 02 14:56:09.974116 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:09.974081    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:10.121456 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:10.121427    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:10.260226 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:10.260144    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:10.446880 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:10.446851    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7g9ww_7c1e0c29-c679-40f0-9961-7f7ff476e04e/node-ca/0.log"
Nov 02 14:56:10.593546 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:10.593468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:10.593546 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:10.593502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:10.594044 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:10.593591    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k67qn" podUID="ce469d24-f9cc-4381-be8a-5c9006c21db7"
Nov 02 14:56:10.594044 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:10.593661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:10.594044 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:10.593761    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5wrn" podUID="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f"
Nov 02 14:56:10.594044 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:10.593824    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l52v6" podUID="35254468-9d2e-4c69-a162-02ae0854b07f"
Nov 02 14:56:12.375914 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.375886    2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeReady"
Nov 02 14:56:12.376237 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.376008    2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 14:56:12.458666 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.458584    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2nhpf"]
Nov 02 14:56:12.482421 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.482395    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jzm4s"]
Nov 02 14:56:12.482550 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.482532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.496775 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.496718    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nhpf"]
Nov 02 14:56:12.496866 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.496798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.498757 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.498733    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:56:12.498757 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.498753    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pj6nr\""
Nov 02 14:56:12.498937 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.498923    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:56:12.507853 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.507831    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jzm4s"]
Nov 02 14:56:12.548824 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.548801    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:56:12.548968 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.548946    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:56:12.549023 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.548972    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:56:12.569641 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.569578    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wfxpz"]
Nov 02 14:56:12.594868 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.594845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:12.598270 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.598247    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wfxpz"]
Nov 02 14:56:12.598363 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.598276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:12.598423 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.598376    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:12.598423 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.598277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:12.599628 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.599613    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-st89p\""
Nov 02 14:56:12.599738 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.599706    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:56:12.610889 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.610871    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:56:12.611368 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.611350    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jmg9z\""
Nov 02 14:56:12.611882 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.611810    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:56:12.611882 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.611858    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:56:12.614755 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.614735    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:12.614833 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.614775    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:12.615061 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.615043    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:56:12.615061 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.615057    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:56:12.615217 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.615108    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:12.615217 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.615129    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:56:12.615217 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.615170    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:12.665850 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.665826    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:12.671907 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.671889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xjr\" (UniqueName: \"kubernetes.io/projected/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-api-access-b7xjr\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.671969 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.671921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bc4f60-d5c6-4390-a61e-7898d781b4c2-metrics-tls\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.671969 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.671940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bc4f60-d5c6-4390-a61e-7898d781b4c2-tmp-dir\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.672045 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.672033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e1b988f-c2c6-4844-84ea-1146294c99b8-crio-socket\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.672079 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.672060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.672156 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.672146    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e1b988f-c2c6-4844-84ea-1146294c99b8-data-volume\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.672189 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.672164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e1b988f-c2c6-4844-84ea-1146294c99b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.672189 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.672181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8kn\" (UniqueName: \"kubernetes.io/projected/67bc4f60-d5c6-4390-a61e-7898d781b4c2-kube-api-access-9s8kn\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.672248 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.672202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bc4f60-d5c6-4390-a61e-7898d781b4c2-config-volume\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.773219 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e1b988f-c2c6-4844-84ea-1146294c99b8-crio-socket\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.773310 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6p6\" (UniqueName: \"kubernetes.io/projected/bbfbe419-497a-489b-9109-c52ed3bf0a6e-kube-api-access-ks6p6\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:12.773310 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.773369 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbfbe419-497a-489b-9109-c52ed3bf0a6e-cert\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:12.773420 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773401    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e1b988f-c2c6-4844-84ea-1146294c99b8-crio-socket\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.773455 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e1b988f-c2c6-4844-84ea-1146294c99b8-data-volume\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.773483 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e1b988f-c2c6-4844-84ea-1146294c99b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.773514 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8kn\" (UniqueName: \"kubernetes.io/projected/67bc4f60-d5c6-4390-a61e-7898d781b4c2-kube-api-access-9s8kn\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.773545 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bc4f60-d5c6-4390-a61e-7898d781b4c2-config-volume\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.773587 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xjr\" (UniqueName: \"kubernetes.io/projected/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-api-access-b7xjr\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.773636 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bc4f60-d5c6-4390-a61e-7898d781b4c2-metrics-tls\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.773718 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bc4f60-d5c6-4390-a61e-7898d781b4c2-tmp-dir\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.773718 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773701    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e1b988f-c2c6-4844-84ea-1146294c99b8-data-volume\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.773983 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.773960    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.774145 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.774126    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bc4f60-d5c6-4390-a61e-7898d781b4c2-config-volume\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.774198 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.774147    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bc4f60-d5c6-4390-a61e-7898d781b4c2-tmp-dir\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.777061 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.777042    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bc4f60-d5c6-4390-a61e-7898d781b4c2-metrics-tls\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.777172 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.777156    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e1b988f-c2c6-4844-84ea-1146294c99b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.798360 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.798333    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xjr\" (UniqueName: \"kubernetes.io/projected/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-api-access-b7xjr\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.798469 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.798453    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8kn\" (UniqueName: \"kubernetes.io/projected/67bc4f60-d5c6-4390-a61e-7898d781b4c2-kube-api-access-9s8kn\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.808045 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.808015    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:12.819732 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.819688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:12.860106 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.860080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:12.874078 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.874050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbfbe419-497a-489b-9109-c52ed3bf0a6e-cert\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:12.874187 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.874133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6p6\" (UniqueName: \"kubernetes.io/projected/bbfbe419-497a-489b-9109-c52ed3bf0a6e-kube-api-access-ks6p6\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:12.877153 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.877130    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbfbe419-497a-489b-9109-c52ed3bf0a6e-cert\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:12.900799 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.900768    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6p6\" (UniqueName: \"kubernetes.io/projected/bbfbe419-497a-489b-9109-c52ed3bf0a6e-kube-api-access-ks6p6\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:12.921046 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:12.921012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:13.176218 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.176193    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nhpf"]
Nov 02 14:56:13.178563 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:13.178530    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bc4f60_d5c6_4390_a61e_7898d781b4c2.slice/crio-3d0ff24a47261e75c9e2bdcc70efcebdb25a58688b5d309bcb5572b46b432a9f WatchSource:0}: Error finding container 3d0ff24a47261e75c9e2bdcc70efcebdb25a58688b5d309bcb5572b46b432a9f: Status 404 returned error can't find the container with id 3d0ff24a47261e75c9e2bdcc70efcebdb25a58688b5d309bcb5572b46b432a9f
Nov 02 14:56:13.179238 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.179216    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jzm4s"]
Nov 02 14:56:13.180389 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:13.180367    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1b988f_c2c6_4844_84ea_1146294c99b8.slice/crio-b7743dc52b8c3cf6888e56afb3a50e50c5094c5bbacf8ecbb6a3e89b250ae0ea WatchSource:0}: Error finding container b7743dc52b8c3cf6888e56afb3a50e50c5094c5bbacf8ecbb6a3e89b250ae0ea: Status 404 returned error can't find the container with id b7743dc52b8c3cf6888e56afb3a50e50c5094c5bbacf8ecbb6a3e89b250ae0ea
Nov 02 14:56:13.220179 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.220160    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wfxpz"]
Nov 02 14:56:13.221556 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:13.221531    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfbe419_497a_489b_9109_c52ed3bf0a6e.slice/crio-03563b5a37e36d6024dbafefac4a9d4a2e88b26070913d492e33be352098375d WatchSource:0}: Error finding container 03563b5a37e36d6024dbafefac4a9d4a2e88b26070913d492e33be352098375d: Status 404 returned error can't find the container with id 03563b5a37e36d6024dbafefac4a9d4a2e88b26070913d492e33be352098375d
Nov 02 14:56:13.421520 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.421485    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:13.870453 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.870208    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wfxpz" event={"ID":"bbfbe419-497a-489b-9109-c52ed3bf0a6e","Type":"ContainerStarted","Data":"03563b5a37e36d6024dbafefac4a9d4a2e88b26070913d492e33be352098375d"}
Nov 02 14:56:13.871969 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.871943    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzm4s" event={"ID":"2e1b988f-c2c6-4844-84ea-1146294c99b8","Type":"ContainerStarted","Data":"9b83ca0e9350830462fa745fdb520b4b161ec9cb8f6698969ec34f15a03bceb1"}
Nov 02 14:56:13.872094 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.871995    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzm4s" event={"ID":"2e1b988f-c2c6-4844-84ea-1146294c99b8","Type":"ContainerStarted","Data":"b7743dc52b8c3cf6888e56afb3a50e50c5094c5bbacf8ecbb6a3e89b250ae0ea"}
Nov 02 14:56:13.873078 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:13.873053    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nhpf" event={"ID":"67bc4f60-d5c6-4390-a61e-7898d781b4c2","Type":"ContainerStarted","Data":"3d0ff24a47261e75c9e2bdcc70efcebdb25a58688b5d309bcb5572b46b432a9f"}
Nov 02 14:56:14.878288 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:14.878247    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzm4s" event={"ID":"2e1b988f-c2c6-4844-84ea-1146294c99b8","Type":"ContainerStarted","Data":"72f6a0599e1e4f76bcac8c860ced114575cdbfe4e58651c9ee03ffd1af06e1f9"}
Nov 02 14:56:14.933912 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:14.933871    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:15.074926 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.074896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:15.280395 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.280302    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:15.448893 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.448853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:15.607468 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.607443    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:15.715751 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.715726    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xhjxz"]
Nov 02 14:56:15.736686 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.736663    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"]
Nov 02 14:56:15.736797 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.736770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.749118 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.748060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:15.749458 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.749390    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"]
Nov 02 14:56:15.755425 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.755290    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:56:15.755425 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.755301    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nts94\""
Nov 02 14:56:15.755425 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.755390    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 14:56:15.760295 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.759916    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 14:56:15.761882 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.761228    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 14:56:15.761882 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.761302    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:56:15.783689 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.783113    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 14:56:15.783689 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.783378    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 14:56:15.784042 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.783987    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 14:56:15.784120 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.784084    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-62xwk\""
Nov 02 14:56:15.785567 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.785542    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:15.794506 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.794449    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"]
Nov 02 14:56:15.809473 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.809441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:15.821558 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.821437    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"]
Nov 02 14:56:15.856022 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.855992    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 14:56:15.856255 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.856234    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 14:56:15.856342 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.856268    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4rlsn\""
Nov 02 14:56:15.856342 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.856316    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 14:56:15.883068 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.883034    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nhpf" event={"ID":"67bc4f60-d5c6-4390-a61e-7898d781b4c2","Type":"ContainerStarted","Data":"b04d6db01fae317ad8854f5c9e8a384d72644b6becd99f3a87857a8b3c3fd408"}
Nov 02 14:56:15.883507 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.883121    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nhpf" event={"ID":"67bc4f60-d5c6-4390-a61e-7898d781b4c2","Type":"ContainerStarted","Data":"e39fb958665a653705c7f4cbc9e79e3bd051300a061946a59f321ab6039ea180"}
Nov 02 14:56:15.883507 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.883180    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:15.899737 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.899712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:15.899871 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.899750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-sys\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.899871 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.899775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-tls\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.899871 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.899802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992dc344-ae2d-43e9-89df-42011f2e8794-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:15.899984 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.899955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5nf\" (UniqueName: \"kubernetes.io/projected/992dc344-ae2d-43e9-89df-42011f2e8794-kube-api-access-2p5nf\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:15.900017 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-textfile\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.900065 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-metrics-client-ca\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.900129 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.900129 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900082    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4tjr\" (UniqueName: \"kubernetes.io/projected/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-kube-api-access-t4tjr\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.900194 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-root\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.900194 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-wtmp\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.900287 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:15.900323 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.900307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:15.925893 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.925864    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:15.953732 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:15.953682    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2nhpf" podStartSLOduration=1.659752229 podStartE2EDuration="3.953663198s" totalImagesPullingTime="2.293910969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:15.953035953 +0000 UTC m=+41.846919209" watchObservedRunningTime="2026-11-02 14:56:15.953663198 +0000 UTC m=+41.847546446"
Nov 02 14:56:16.001876 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.001814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-textfile\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.001876 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.001863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ea0a743-d6e8-4211-bd0f-b719dfc91508-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.002091 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.001917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4tjr\" (UniqueName: \"kubernetes.io/projected/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-kube-api-access-t4tjr\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002091 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.001945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-root\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002091 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.001973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.002091 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-tls\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002211 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992dc344-ae2d-43e9-89df-42011f2e8794-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.002211 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-metrics-client-ca\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002211 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-wtmp\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002296 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002296 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.002350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq9g8\" (UniqueName: \"kubernetes.io/projected/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-api-access-lq9g8\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.002350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.002350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.002432 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.002432 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.002495 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-sys\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002495 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5nf\" (UniqueName: \"kubernetes.io/projected/992dc344-ae2d-43e9-89df-42011f2e8794-kube-api-access-2p5nf\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.002495 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002783 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002163    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-textfile\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.002890 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.002850    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-wtmp\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.003162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.003137    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.003239 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.003203    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-root\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.003301 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.003261    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992dc344-ae2d-43e9-89df-42011f2e8794-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.003357 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.003319    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-metrics-client-ca\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.003479 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.003459    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-sys\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.006383 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.006359    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.006496 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.006363    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.006558 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.006523    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-tls\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.006629 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.006612    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.030680 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.030638    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4tjr\" (UniqueName: \"kubernetes.io/projected/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-kube-api-access-t4tjr\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.035119 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.035093    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5nf\" (UniqueName: \"kubernetes.io/projected/992dc344-ae2d-43e9-89df-42011f2e8794-kube-api-access-2p5nf\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.079884 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.079813    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:16.103657 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.103618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.103790 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.103672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9g8\" (UniqueName: \"kubernetes.io/projected/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-api-access-lq9g8\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.103790 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.103703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.103790 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.103732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.103790 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.103761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.103992 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.103812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ea0a743-d6e8-4211-bd0f-b719dfc91508-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.104609 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.104582    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.104834 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.104816    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ea0a743-d6e8-4211-bd0f-b719dfc91508-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.105237 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.105203    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.106578 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.106557    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.106578 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.106570    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.116426 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.116386    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:16.121555 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:16.121528    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14a5cf5_20d5_47c5_bdf8_bccfe1ffc6f3.slice/crio-d2ef7f7f77387e2b61fa8a1bb4b397b2469cc2bf80f8c50e3a16675311135aab WatchSource:0}: Error finding container d2ef7f7f77387e2b61fa8a1bb4b397b2469cc2bf80f8c50e3a16675311135aab: Status 404 returned error can't find the container with id d2ef7f7f77387e2b61fa8a1bb4b397b2469cc2bf80f8c50e3a16675311135aab
Nov 02 14:56:16.129586 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.129566    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9g8\" (UniqueName: \"kubernetes.io/projected/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-api-access-lq9g8\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.131437 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.131412    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:16.174483 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.174241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:16.278943 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.278909    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:16.454088 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.454064    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"]
Nov 02 14:56:16.497190 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.497167    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"]
Nov 02 14:56:16.506819 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:16.506794    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992dc344_ae2d_43e9_89df_42011f2e8794.slice/crio-ec960c0e56baa2c6e9dd89ae7ed8e1b3d02cc6af317a3055d6baebb5e1d94304 WatchSource:0}: Error finding container ec960c0e56baa2c6e9dd89ae7ed8e1b3d02cc6af317a3055d6baebb5e1d94304: Status 404 returned error can't find the container with id ec960c0e56baa2c6e9dd89ae7ed8e1b3d02cc6af317a3055d6baebb5e1d94304
Nov 02 14:56:16.507390 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:16.507361    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea0a743_d6e8_4211_bd0f_b719dfc91508.slice/crio-fdb804e389c36a44ed5d20fdd4cf6b6f6cc0f08279f6407758eeddd62a71768b WatchSource:0}: Error finding container fdb804e389c36a44ed5d20fdd4cf6b6f6cc0f08279f6407758eeddd62a71768b: Status 404 returned error can't find the container with id fdb804e389c36a44ed5d20fdd4cf6b6f6cc0f08279f6407758eeddd62a71768b
Nov 02 14:56:16.854064 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.854026    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:16.887707 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.887489    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wfxpz" event={"ID":"bbfbe419-497a-489b-9109-c52ed3bf0a6e","Type":"ContainerStarted","Data":"29e7cb5ee84950ce10ab8f302eb30f47574f7c0a3cd7b77549d844134820bf9a"}
Nov 02 14:56:16.889209 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.889186    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhjxz" event={"ID":"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3","Type":"ContainerStarted","Data":"d2ef7f7f77387e2b61fa8a1bb4b397b2469cc2bf80f8c50e3a16675311135aab"}
Nov 02 14:56:16.890563 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.890538    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh" event={"ID":"8ea0a743-d6e8-4211-bd0f-b719dfc91508","Type":"ContainerStarted","Data":"fdb804e389c36a44ed5d20fdd4cf6b6f6cc0f08279f6407758eeddd62a71768b"}
Nov 02 14:56:16.892766 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.892742    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6" event={"ID":"992dc344-ae2d-43e9-89df-42011f2e8794","Type":"ContainerStarted","Data":"6220ddcc49d0deab4f14d2c09a960aa332c831f46cdfede87f065c698f4226e7"}
Nov 02 14:56:16.892852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.892796    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6" event={"ID":"992dc344-ae2d-43e9-89df-42011f2e8794","Type":"ContainerStarted","Data":"f5ccb0e3b3c6fe8a6ab7560684c59c595334b0a94c3d3080bce03d1cb1555d00"}
Nov 02 14:56:16.892852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.892828    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6" event={"ID":"992dc344-ae2d-43e9-89df-42011f2e8794","Type":"ContainerStarted","Data":"ec960c0e56baa2c6e9dd89ae7ed8e1b3d02cc6af317a3055d6baebb5e1d94304"}
Nov 02 14:56:16.931895 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:16.931847    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wfxpz" podStartSLOduration=1.61176869 podStartE2EDuration="4.931830511s" totalImagesPullingTime="3.320061821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:16.931334465 +0000 UTC m=+42.825217718" watchObservedRunningTime="2026-11-02 14:56:16.931830511 +0000 UTC m=+42.825713767"
Nov 02 14:56:17.002810 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.002721    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:17.129165 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.129132    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:17.131944 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.131908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.152850 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.152751    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 14:56:17.153058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.153042    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 14:56:17.153489 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.153470    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 14:56:17.153806 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.153790    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 14:56:17.154043 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.154021    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 14:56:17.154445 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.154425    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 14:56:17.155707 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.154762    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 14:56:17.155707 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.154819    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 14:56:17.155707 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.155029    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hckbz\""
Nov 02 14:56:17.155707 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.155142    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 14:56:17.165923 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.165896    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:17.167305 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.167287    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:17.214052 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.213970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214052 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214261 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214261 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214261 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5b4q\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214261 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214230    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214261 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214441 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214441 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214441 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214441 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214441 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.214441 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.214439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.314948 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.314904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315062 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.314961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315062 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315062 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5b4q\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315213 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315213 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315213 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315213 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315386 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315386 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315386 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315386 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.315386 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.315332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.316066 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.316043    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.316678 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:17.316321    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle podName:ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:17.816297167 +0000 UTC m=+43.710180413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231") : configmap references non-existent config key: ca-bundle.crt
Nov 02 14:56:17.316678 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.316593    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.320307 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.320282    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.320465 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.320447    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.320543 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.320466    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.320749 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.320727    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.320913 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.320897    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.320974 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.320915    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.321462 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.321439    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.321462 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.321449    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.321944 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.321921    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.340546 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.340521    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5b4q\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.819304 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.819269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:17.819482 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:17.819457    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle podName:ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:18.819432821 +0000 UTC m=+44.713316059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231") : configmap references non-existent config key: ca-bundle.crt
Nov 02 14:56:17.898801 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.898769    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzm4s" event={"ID":"2e1b988f-c2c6-4844-84ea-1146294c99b8","Type":"ContainerStarted","Data":"63f538e05ca0012aaf16fdbc25aac84e799faf009920339e29e426a85d76fa2a"}
Nov 02 14:56:17.900597 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.900572    2581 generic.go:356] "Generic (PLEG): container finished" podID="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" containerID="2290f6905a8abf97609fa8b8538552a503cb907b11af86286f65547215060b38" exitCode=0
Nov 02 14:56:17.900723 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.900677    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhjxz" event={"ID":"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3","Type":"ContainerDied","Data":"2290f6905a8abf97609fa8b8538552a503cb907b11af86286f65547215060b38"}
Nov 02 14:56:17.918887 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.918862    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6895fc499c-x7njf"]
Nov 02 14:56:17.923859 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.923838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:17.934679 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.934640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:17.958053 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.958031    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 14:56:17.958158 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.958122    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5g5zn\""
Nov 02 14:56:17.958206 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.958168    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 14:56:17.958252 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.958235    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 14:56:17.961541 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.959638    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jzm4s" podStartSLOduration=2.469219162 podStartE2EDuration="5.959621262s" totalImagesPullingTime="3.4904021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:17.958394989 +0000 UTC m=+43.852278245" watchObservedRunningTime="2026-11-02 14:56:17.959621262 +0000 UTC m=+43.853504517"
Nov 02 14:56:17.961969 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:17.961942    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6895fc499c-x7njf"]
Nov 02 14:56:18.022498 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022563 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022563 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-tmp\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022563 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022681 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022681 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5t4\" (UniqueName: \"kubernetes.io/projected/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-kube-api-access-dt5t4\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022681 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-grpc-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022789 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.022894 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.022871    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-metrics-client-ca\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.049626 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.049302    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 14:56:18.049626 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.049423    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1hearnjc8vakj\""
Nov 02 14:56:18.049626 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.049545    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 14:56:18.093411 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.093380    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-metrics-client-ca\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-tmp\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5t4\" (UniqueName: \"kubernetes.io/projected/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-kube-api-access-dt5t4\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-grpc-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.125019 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.124484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.126493 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.126464    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-tmp\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.127123 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.127078    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-metrics-client-ca\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.128208 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.128191    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.128350 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.128304    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.128501 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.128485    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.128696 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.128679    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-grpc-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.128873 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.128854    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.130804 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.130784    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.160234 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.160164    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5t4\" (UniqueName: \"kubernetes.io/projected/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-kube-api-access-dt5t4\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.274895 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.274869    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:18.601790 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.601580    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6895fc499c-x7njf"]
Nov 02 14:56:18.602723 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:18.602701    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34c8a6c_4225_4d6f_9886_9f1f908d2f5a.slice/crio-0bdfd6fce571361dd95f339233f628494ba762123efb20054033f5b8aecfd242 WatchSource:0}: Error finding container 0bdfd6fce571361dd95f339233f628494ba762123efb20054033f5b8aecfd242: Status 404 returned error can't find the container with id 0bdfd6fce571361dd95f339233f628494ba762123efb20054033f5b8aecfd242
Nov 02 14:56:18.832275 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.832240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:18.833099 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.833079    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:18.907698 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.907643    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" event={"ID":"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a","Type":"ContainerStarted","Data":"0bdfd6fce571361dd95f339233f628494ba762123efb20054033f5b8aecfd242"}
Nov 02 14:56:18.909612 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.909591    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhjxz" event={"ID":"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3","Type":"ContainerStarted","Data":"914e9bb45e79a3957830099cc592b946ddad33907544c700aad62a5b9fea7192"}
Nov 02 14:56:18.909754 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.909633    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xhjxz" event={"ID":"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3","Type":"ContainerStarted","Data":"8f18f82ef09d5a37f1d9718bc1ecb0d89643443355fdc14b5e6059f5dc77861b"}
Nov 02 14:56:18.911541 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.911523    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh" event={"ID":"8ea0a743-d6e8-4211-bd0f-b719dfc91508","Type":"ContainerStarted","Data":"7e77ecf1916b337ec796a06389c9a0b9bb8c343dad798ce522850473ee0b8087"}
Nov 02 14:56:18.911622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.911564    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh" event={"ID":"8ea0a743-d6e8-4211-bd0f-b719dfc91508","Type":"ContainerStarted","Data":"edcc1074688bf255b02395ac818d5da875f430d26225fa8927a52b3ee4f4368a"}
Nov 02 14:56:18.911622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.911595    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh" event={"ID":"8ea0a743-d6e8-4211-bd0f-b719dfc91508","Type":"ContainerStarted","Data":"be0de8e2e98765729345986b557526d9f6a11851d4a09665e44c6022dbe0c24a"}
Nov 02 14:56:18.913432 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.913411    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6" event={"ID":"992dc344-ae2d-43e9-89df-42011f2e8794","Type":"ContainerStarted","Data":"4900056e04dbfa39559a9a0efe77481b2860361130e022a74f3f9811bc993582"}
Nov 02 14:56:18.953537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.953513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:18.957175 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.957133    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:18.957803 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:18.957766    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xhjxz" podStartSLOduration=2.686314204 podStartE2EDuration="3.957756211s" totalImagesPullingTime="1.18547626s" totalInitContainerRuntime="85.965747ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:18.957455829 +0000 UTC m=+44.851339083" watchObservedRunningTime="2026-11-02 14:56:18.957756211 +0000 UTC m=+44.851639462"
Nov 02 14:56:19.008769 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:19.006810    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6" podStartSLOduration=3.034220599 podStartE2EDuration="4.006791934s" totalImagesPullingTime="972.571335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:19.005772227 +0000 UTC m=+44.899655480" watchObservedRunningTime="2026-11-02 14:56:19.006791934 +0000 UTC m=+44.900675190"
Nov 02 14:56:19.052383 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:19.052337    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh" podStartSLOduration=2.696242601 podStartE2EDuration="4.052327386s" totalImagesPullingTime="1.356084785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:19.052294516 +0000 UTC m=+44.946177772" watchObservedRunningTime="2026-11-02 14:56:19.052327386 +0000 UTC m=+44.946210635"
Nov 02 14:56:19.110429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:19.110393    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:19.246031 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:19.245955    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:19.274236 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:19.274212    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:19.920344 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:19.920309    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerStarted","Data":"3facf7fca9c04659b8c62219dcff6e59452a1817f042f3e4f6530ed03886ac26"}
Nov 02 14:56:20.239308 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.239186    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64b454d547-2rcbx"]
Nov 02 14:56:20.241287 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.241264    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.263212 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.263184    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 14:56:20.271201 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.271172    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 14:56:20.272735 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.272712    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64b454d547-2rcbx"]
Nov 02 14:56:20.328354 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.328327    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 14:56:20.329020 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.328904    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4qhtv\""
Nov 02 14:56:20.329097 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.329052    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aha88bqjs45nv\""
Nov 02 14:56:20.329156 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.329130    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 14:56:20.344838 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.344810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-client-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.344950 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.344891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-metrics-server-audit-profiles\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.344950 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.344918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wt7\" (UniqueName: \"kubernetes.io/projected/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-kube-api-access-t7wt7\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.345059 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.344952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.345059 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.344981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-client-certs\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.345059 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.345003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-audit-log\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.345059 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.345027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-tls\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.446278 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.446243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-tls\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.446467 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.446359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-client-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.446467 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.446437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-metrics-server-audit-profiles\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.446582 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.446467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wt7\" (UniqueName: \"kubernetes.io/projected/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-kube-api-access-t7wt7\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.446582 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.446504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.446767 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.446703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-client-certs\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.447152 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.446937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-audit-log\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.447313 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.447291    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-audit-log\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.447383 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.447359    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.448022 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.447976    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-metrics-server-audit-profiles\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.449514 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.449487    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-client-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.449609 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.449589    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-tls\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.450100 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.450058    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-client-certs\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.471616 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.471587    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wt7\" (UniqueName: \"kubernetes.io/projected/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-kube-api-access-t7wt7\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.494813 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.494748    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:20.517232 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.517212    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"]
Nov 02 14:56:20.519852 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.519836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.541780 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.540889    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 14:56:20.541780 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.541038    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dlg54\""
Nov 02 14:56:20.542532 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.542511    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"]
Nov 02 14:56:20.548081 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.548057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv26p\" (UniqueName: \"kubernetes.io/projected/be62e46e-7648-4e29-a2ba-fce4655087ee-kube-api-access-nv26p\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.548214 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.548171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be62e46e-7648-4e29-a2ba-fce4655087ee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.555809 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.555788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:20.648855 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.648831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv26p\" (UniqueName: \"kubernetes.io/projected/be62e46e-7648-4e29-a2ba-fce4655087ee-kube-api-access-nv26p\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.648945 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.648884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be62e46e-7648-4e29-a2ba-fce4655087ee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.652132 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.651770    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be62e46e-7648-4e29-a2ba-fce4655087ee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.652132 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.651965    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:20.674214 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.674195    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv26p\" (UniqueName: \"kubernetes.io/projected/be62e46e-7648-4e29-a2ba-fce4655087ee-kube-api-access-nv26p\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.839857 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.839782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:20.859640 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.859612    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:20.867971 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.867944    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64b454d547-2rcbx"]
Nov 02 14:56:20.870797 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:20.870769    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3e864f_07ce_47ff_a6f5_e4197c15f72d.slice/crio-b3db888c9ffb7cc0201660ff7b957961dc033e25b5075dcc7e41ab27a52f6e73 WatchSource:0}: Error finding container b3db888c9ffb7cc0201660ff7b957961dc033e25b5075dcc7e41ab27a52f6e73: Status 404 returned error can't find the container with id b3db888c9ffb7cc0201660ff7b957961dc033e25b5075dcc7e41ab27a52f6e73
Nov 02 14:56:20.925883 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.925855    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" event={"ID":"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a","Type":"ContainerStarted","Data":"32a008e697931b96f4464a630984bf2a4b91ee1ee7a16dbe0b74b630bfb154cc"}
Nov 02 14:56:20.926260 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.925920    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" event={"ID":"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a","Type":"ContainerStarted","Data":"5f67648107965f7acc34ccf2bccc979d354369185a6e8ad831690011627af8cd"}
Nov 02 14:56:20.926260 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.925949    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" event={"ID":"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a","Type":"ContainerStarted","Data":"5c0068ea85484c45732ccc06dccec53da097383400b77025e5848ee469f7a804"}
Nov 02 14:56:20.927311 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.927289    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675" exitCode=0
Nov 02 14:56:20.927415 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.927376    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"}
Nov 02 14:56:20.928572 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.928551    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx" event={"ID":"3d3e864f-07ce-47ff-a6f5-e4197c15f72d","Type":"ContainerStarted","Data":"b3db888c9ffb7cc0201660ff7b957961dc033e25b5075dcc7e41ab27a52f6e73"}
Nov 02 14:56:20.992183 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.992156    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"]
Nov 02 14:56:20.995213 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:20.995195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.031581 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.031538    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 14:56:21.045537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.045226    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 14:56:21.045537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.045325    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 14:56:21.045537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.045362    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 14:56:21.045537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.045392    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 14:56:21.045537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.045427    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 14:56:21.045537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.045508    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wnnhb\""
Nov 02 14:56:21.052772 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.049284    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"]
Nov 02 14:56:21.054747 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.054720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.054866 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.054848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-serving-certs-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.054957 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.054937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.055058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.054984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-metrics-client-ca\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.055111 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.055064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-federate-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.055111 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.055096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqc8c\" (UniqueName: \"kubernetes.io/projected/11e1276c-fe6b-44e7-8df9-a7925a8cd350-kube-api-access-tqc8c\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.055186 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.055131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.055186 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.055158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.145589 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.145557    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"]
Nov 02 14:56:21.146521 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:21.146451    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe62e46e_7648_4e29_a2ba_fce4655087ee.slice/crio-09dbf9b29b2967d6df9539f0392e68195a5042b74c351ffa037ac7948da66fba WatchSource:0}: Error finding container 09dbf9b29b2967d6df9539f0392e68195a5042b74c351ffa037ac7948da66fba: Status 404 returned error can't find the container with id 09dbf9b29b2967d6df9539f0392e68195a5042b74c351ffa037ac7948da66fba
Nov 02 14:56:21.156220 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-federate-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.156310 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqc8c\" (UniqueName: \"kubernetes.io/projected/11e1276c-fe6b-44e7-8df9-a7925a8cd350-kube-api-access-tqc8c\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.156310 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.156428 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.156428 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.156540 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-serving-certs-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.156540 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.158842 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.156538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-metrics-client-ca\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.158842 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.157366    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-metrics-client-ca\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.158842 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.157634    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.158842 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.158190    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-serving-certs-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.159222 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.159201    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-federate-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.159603 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.159585    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.159805 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.159784    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.160961 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.160927    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.182791 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.182764    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqc8c\" (UniqueName: \"kubernetes.io/projected/11e1276c-fe6b-44e7-8df9-a7925a8cd350-kube-api-access-tqc8c\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.307137 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.306889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:21.635427 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.635389    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"]
Nov 02 14:56:21.637417 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:21.637393    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e1276c_fe6b_44e7_8df9_a7925a8cd350.slice/crio-8d7372812047d0ba0fb5c170b6ee62d0719e2a65f6477705b8b9647d3ba9a7cf WatchSource:0}: Error finding container 8d7372812047d0ba0fb5c170b6ee62d0719e2a65f6477705b8b9647d3ba9a7cf: Status 404 returned error can't find the container with id 8d7372812047d0ba0fb5c170b6ee62d0719e2a65f6477705b8b9647d3ba9a7cf
Nov 02 14:56:21.938485 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.938452    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" event={"ID":"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a","Type":"ContainerStarted","Data":"9d261deb6c7789196531b345c15cf4c567cd767e3bc119aa2cc42f69431aef09"}
Nov 02 14:56:21.938976 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.938535    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" event={"ID":"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a","Type":"ContainerStarted","Data":"5f17be558f926bfc77c8466461b5c9fff2d1dac0d2d98bdd98598bff24cb6ad3"}
Nov 02 14:56:21.938976 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.938573    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" event={"ID":"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a","Type":"ContainerStarted","Data":"c3d9883517f5b363a107d353ce015b44d1a4cf70e0dc5f34c1590ed92fdbc709"}
Nov 02 14:56:21.939090 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.939057    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:21.940304 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.940274    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn" event={"ID":"11e1276c-fe6b-44e7-8df9-a7925a8cd350","Type":"ContainerStarted","Data":"8d7372812047d0ba0fb5c170b6ee62d0719e2a65f6477705b8b9647d3ba9a7cf"}
Nov 02 14:56:21.942194 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.942174    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9" event={"ID":"be62e46e-7648-4e29-a2ba-fce4655087ee","Type":"ContainerStarted","Data":"09dbf9b29b2967d6df9539f0392e68195a5042b74c351ffa037ac7948da66fba"}
Nov 02 14:56:21.994991 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:21.994930    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf" podStartSLOduration=2.413820862 podStartE2EDuration="4.994913095s" totalImagesPullingTime="2.581092233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:21.994207764 +0000 UTC m=+47.888091019" watchObservedRunningTime="2026-11-02 14:56:21.994913095 +0000 UTC m=+47.888796350"
Nov 02 14:56:22.666689 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.664443    2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:56:22.669050 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.669005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.717855 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.717822    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:22.721091 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.721067    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 14:56:22.721205 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.721176    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 14:56:22.721258 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.721214    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 14:56:22.721258 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.721228    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 14:56:22.721370 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.721285    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 14:56:22.721370 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.721306    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 14:56:22.721470 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.721372    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 14:56:22.723501 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.723475    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 14:56:22.725393 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.725362    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:56:22.734861 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.734464    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 14:56:22.734861 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.734519    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 14:56:22.734861 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.734571    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 14:56:22.734861 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.734622    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 14:56:22.734861 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.734680    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eqr3qis59nkhr\""
Nov 02 14:56:22.734861 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.734773    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f8vff\""
Nov 02 14:56:22.734861 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.734830    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 14:56:22.735387 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.735371    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 14:56:22.770823 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.770907 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.770907 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.770907 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.770977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771058 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771076    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfj5v\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.771416 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.771214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872311 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872286    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfj5v\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872466    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.872897 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.872643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.876622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.876410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.876622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.876484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.876622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.874584    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.876622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.875374    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.876622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.875535    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.876622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.875726    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.876622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.876343    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.877392 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.873698    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.877911 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.877606    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.877911 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.873851    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.877911 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.877680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.877911 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.877732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.877911 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.877758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.878498 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.878471    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.881020 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.880990    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.881106 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.881040    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.881162 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.881146    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.882009 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.881472    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.882009 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.881851    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.882732 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.882677    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.882851 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.882640    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.883170 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.883150    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.884389 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.884368    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.884389 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.884393    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.901840 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.899495    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfj5v\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:22.947496 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.947469    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9" event={"ID":"be62e46e-7648-4e29-a2ba-fce4655087ee","Type":"ContainerStarted","Data":"09ca74b8d9e96b4cd834ebbddfb4c3f7d60e651bb00febae5606398fa789f80e"}
Nov 02 14:56:22.947923 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.947678    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:22.952771 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.952732    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerStarted","Data":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"}
Nov 02 14:56:22.952899 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.952794    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerStarted","Data":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"}
Nov 02 14:56:22.952899 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.952821    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerStarted","Data":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"}
Nov 02 14:56:22.954042 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.954020    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:22.955202 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.955179    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx" event={"ID":"3d3e864f-07ce-47ff-a6f5-e4197c15f72d","Type":"ContainerStarted","Data":"7884da21652b2e41f4646567198de00c96639622f5726c289ace64c09855d76a"}
Nov 02 14:56:22.995197 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:22.995156    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9" podStartSLOduration=1.509526213 podStartE2EDuration="2.995139553s" totalImagesPullingTime="1.48561334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:22.992381652 +0000 UTC m=+48.886264907" watchObservedRunningTime="2026-11-02 14:56:22.995139553 +0000 UTC m=+48.889022807"
Nov 02 14:56:23.045622 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.045597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:23.086302 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.086042    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx" podStartSLOduration=1.37429464 podStartE2EDuration="3.086027089s" totalImagesPullingTime="1.711732449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:23.085930573 +0000 UTC m=+48.979813828" watchObservedRunningTime="2026-11-02 14:56:23.086027089 +0000 UTC m=+48.979910344"
Nov 02 14:56:23.406812 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.405065    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:56:23.416570 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:23.416541    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18cfe9b9_6abc_494a_aa25_201041f17bd1.slice/crio-6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc WatchSource:0}: Error finding container 6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc: Status 404 returned error can't find the container with id 6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc
Nov 02 14:56:23.960662 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.960610    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn" event={"ID":"11e1276c-fe6b-44e7-8df9-a7925a8cd350","Type":"ContainerStarted","Data":"35e6f90cec6e6cbe5d3d12eea0164376b441d78164b0d80057fe3d2b48c42525"}
Nov 02 14:56:23.961146 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.960690    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn" event={"ID":"11e1276c-fe6b-44e7-8df9-a7925a8cd350","Type":"ContainerStarted","Data":"dc3cf15bc201143f074e0d5fb41d8fff6f3eb4755531498b2b6fb2da91f7e238"}
Nov 02 14:56:23.961146 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.960720    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn" event={"ID":"11e1276c-fe6b-44e7-8df9-a7925a8cd350","Type":"ContainerStarted","Data":"7e4dd4c52c6b1dfe5dbc224d8bf97e4d0dbfd840fdc216b06de62ba26a456af0"}
Nov 02 14:56:23.963957 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.963934    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerStarted","Data":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"}
Nov 02 14:56:23.964076 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.963998    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerStarted","Data":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"}
Nov 02 14:56:23.964076 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.964033    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerStarted","Data":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"}
Nov 02 14:56:23.965569 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.965546    2581 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7" exitCode=0
Nov 02 14:56:23.965684 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.965632    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"}
Nov 02 14:56:23.965752 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:23.965708    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerStarted","Data":"6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc"}
Nov 02 14:56:24.016050 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:24.015972    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn" podStartSLOduration=1.919910663 podStartE2EDuration="4.015957805s" totalImagesPullingTime="2.096047142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:24.014418143 +0000 UTC m=+49.908301397" watchObservedRunningTime="2026-11-02 14:56:24.015957805 +0000 UTC m=+49.909841071"
Nov 02 14:56:24.073625 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:24.073586    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.321301054 podStartE2EDuration="7.073575626s" totalImagesPullingTime="2.733512216s" totalInitContainerRuntime="18.762356ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:24.07251753 +0000 UTC m=+49.966400788" watchObservedRunningTime="2026-11-02 14:56:24.073575626 +0000 UTC m=+49.967458904"
Nov 02 14:56:25.182433 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:25.182396    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:25.331893 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:25.331853    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:25.719912 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:25.719880    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:25.878119 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:25.878088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:25.895811 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:25.895785    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:26.984304 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:26.984279    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerStarted","Data":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"}
Nov 02 14:56:27.963805 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:27.963779    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:27.992131 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:27.992103    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerStarted","Data":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"}
Nov 02 14:56:27.992471 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:27.992169    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerStarted","Data":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"}
Nov 02 14:56:27.992471 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:27.992196    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerStarted","Data":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"}
Nov 02 14:56:27.992471 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:27.992220    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerStarted","Data":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"}
Nov 02 14:56:27.992471 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:27.992247    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerStarted","Data":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"}
Nov 02 14:56:28.047756 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:28.047725    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:28.092827 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:28.092773    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.154340624 podStartE2EDuration="6.092759159s" totalImagesPullingTime="2.92219992s" totalInitContainerRuntime="16.218615ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:28.091067766 +0000 UTC m=+53.984951043" watchObservedRunningTime="2026-11-02 14:56:28.092759159 +0000 UTC m=+53.986642446"
Nov 02 14:56:28.275017 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:28.274913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:28.441130 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:28.441096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:28.861260 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:28.861235    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:29.360529 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:29.360454    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:29.562811 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:29.562771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:31.838369 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:31.838331    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:32.016287 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:32.016248    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:32.533862 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:32.533832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:32.673096 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:32.673065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:32.818159 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:32.818079    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:33.544780 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:33.544744    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:33.705491 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:33.705456    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:35.328438 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:35.328402    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:35.477748 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:35.477715    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:35.643203 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:35.643168    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:35.790885 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:35.790848    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:37.276498 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:37.276464    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:37.480663 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:37.480628    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:37.630641 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:37.630614    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:40.348869 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.348838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:40.349275 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.348908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:40.364442 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.364411    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:40.371429 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.371403    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:40.377964 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.377947    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:40.381897 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.381874    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:40.449505 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.449475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:40.464948 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.464925    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:40.485442 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.485425    2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:40.493068 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.493042    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:40.550304 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.550234    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:56:40.554537 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.554511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:40.556100 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.556080    2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:40.556222 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.556173    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:40.563472 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.563452    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:56:40.568122 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.568103    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:40.589558 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.589529    2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:56:40.595027 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.594793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:40.850470 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.850445    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k67qn"]
Nov 02 14:56:40.857358 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:40.857314    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce469d24_f9cc_4381_be8a_5c9006c21db7.slice/crio-fb0b738f1485ce90d6752e9394772345c13ca0be9db2dd061e008189ddd89ae1 WatchSource:0}: Error finding container fb0b738f1485ce90d6752e9394772345c13ca0be9db2dd061e008189ddd89ae1: Status 404 returned error can't find the container with id fb0b738f1485ce90d6752e9394772345c13ca0be9db2dd061e008189ddd89ae1
Nov 02 14:56:40.867996 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.867968    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l52v6"]
Nov 02 14:56:40.868329 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:40.868308    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35254468_9d2e_4c69_a162_02ae0854b07f.slice/crio-ff493e3e73dd9b01267f73c40b92683d80615c0994558b5b3df4e1af910f9f67 WatchSource:0}: Error finding container ff493e3e73dd9b01267f73c40b92683d80615c0994558b5b3df4e1af910f9f67: Status 404 returned error can't find the container with id ff493e3e73dd9b01267f73c40b92683d80615c0994558b5b3df4e1af910f9f67
Nov 02 14:56:40.926901 ip-10-0-129-45 kubenswrapper[2581]: W1102 14:56:40.926874    2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1313c9_5c99_44a9_914f_3bdbcfd9a23f.slice/crio-32bef338037d22e11dba9febf7f048f54e4ee84110c8620b305c99ca7c7f0501 WatchSource:0}: Error finding container 32bef338037d22e11dba9febf7f048f54e4ee84110c8620b305c99ca7c7f0501: Status 404 returned error can't find the container with id 32bef338037d22e11dba9febf7f048f54e4ee84110c8620b305c99ca7c7f0501
Nov 02 14:56:40.927435 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:40.927417    2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5wrn"]
Nov 02 14:56:41.045371 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:41.045345    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k67qn" event={"ID":"ce469d24-f9cc-4381-be8a-5c9006c21db7","Type":"ContainerStarted","Data":"fb0b738f1485ce90d6752e9394772345c13ca0be9db2dd061e008189ddd89ae1"}
Nov 02 14:56:41.046531 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:41.046502    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5wrn" event={"ID":"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f","Type":"ContainerStarted","Data":"32bef338037d22e11dba9febf7f048f54e4ee84110c8620b305c99ca7c7f0501"}
Nov 02 14:56:41.047693 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:41.047669    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l52v6" event={"ID":"35254468-9d2e-4c69-a162-02ae0854b07f","Type":"ContainerStarted","Data":"ff493e3e73dd9b01267f73c40b92683d80615c0994558b5b3df4e1af910f9f67"}
Nov 02 14:56:42.888766 ip-10-0-129-45 kubenswrapper[2581]: E1102 14:56:42.888722    2581 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:56:43.058984 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:43.058955    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5wrn" event={"ID":"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f","Type":"ContainerStarted","Data":"bb2aba7665e6b1a905baeaed6cec475dda29db12d0a35191593d4aeccf058aba"}
Nov 02 14:56:43.059160 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:43.059017    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5wrn" event={"ID":"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f","Type":"ContainerStarted","Data":"48ed187f78f2e060b0dcffeebabb46d4cb046e01042502933550903673000f6e"}
Nov 02 14:56:43.105596 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:43.105533    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h5wrn" podStartSLOduration=68.061741682 podStartE2EDuration="1m9.105515125s" totalImagesPullingTime="1.043773443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:43.10400945 +0000 UTC m=+68.997892705" watchObservedRunningTime="2026-11-02 14:56:43.105515125 +0000 UTC m=+68.999398382"
Nov 02 14:56:43.894454 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:43.894419    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:44.039876 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:44.039792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:45.070035 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:45.069488    2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k67qn" event={"ID":"ce469d24-f9cc-4381-be8a-5c9006c21db7","Type":"ContainerStarted","Data":"9d121e3511a631e7b0137eb5d14023093403cc0ed8560d1f8d16546b3e3c5faf"}
Nov 02 14:56:45.070035 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:45.069627    2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:45.113249 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:45.113198    2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k67qn" podStartSLOduration=67.80210303 podStartE2EDuration="1m11.113184695s" totalImagesPullingTime="3.311081665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:45.112780524 +0000 UTC m=+71.006663779" watchObservedRunningTime="2026-11-02 14:56:45.113184695 +0000 UTC m=+71.007067948"
Nov 02 14:56:45.657882 ip-10-0-129-45 kubenswrapper[2581]: I1102 14:56:45.657789    2581 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:56:45.657955 ip-10-0-129-45 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 14:56:45.679997 ip-10-0-129-45 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 14:56:45.680163 ip-10-0-129-45 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 14:56:45.680338 ip-10-0-129-45 systemd[1]: kubelet.service: Consumed 4.748s CPU time.
Nov 02 14:56:45.694346 ip-10-0-129-45 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 14:56:45.774985 ip-10-0-129-45 kubenswrapper[7019]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:45.774985 ip-10-0-129-45 kubenswrapper[7019]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 14:56:45.774985 ip-10-0-129-45 kubenswrapper[7019]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:45.774985 ip-10-0-129-45 kubenswrapper[7019]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 14:56:45.777237 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777159    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.777237 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777236    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777241    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777244    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777248    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777251    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777262    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777265    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777267    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777271    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777274    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777277    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777280    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777282    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777286    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777288    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777291    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777294    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777297    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777299    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777302    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.777311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777305    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777308    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777310    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777313    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777316    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777319    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777322    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777325    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777328    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777330    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777333    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777336    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777338    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777341    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777343    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777346    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777349    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777351    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777354    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777357    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.777829 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777359    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777362    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777364    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777367    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777370    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777373    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777376    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777378    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777381    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777383    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777386    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777393    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777396    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777398    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777402    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777405    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777407    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777410    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777413    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777415    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.778323 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777418    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777421    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777424    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777428    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777432    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777435    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777440    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777443    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777445    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777450    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777454    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777458    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777460    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777463    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777466    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777469    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777472    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777475    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777477    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.778821 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777480    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777482    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777485    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777488    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777492    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777495    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777497    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777500    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777503    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777505    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777509    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777511    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777515    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777517    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777521    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777525    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777529    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777532    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777535    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.779294 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777537    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777542    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777546    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777549    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777552    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777555    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.777558    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778297    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778304    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778308    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778311    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778314    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778316    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778319    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778322    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778325    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778327    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778332    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778336    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.779803 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778339    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778342    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778344    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778347    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778351    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778355    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778359    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778363    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778366    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778369    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778372    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778374    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778377    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778380    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778382    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778385    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778388    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778392    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778395    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.780272 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778397    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778406    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778409    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778412    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778415    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778418    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778420    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778423    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778425    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778434    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778437    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778441    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778444    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778448    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778451    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778453    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778456    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778458    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778461    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778464    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.780769 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778467    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778469    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778472    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778475    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778477    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778481    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778485    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778487    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778491    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778494    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778498    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778500    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778503    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778506    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778516    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778519    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778522    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778524    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778527    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.781260 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778529    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778532    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778535    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778537    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778541    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778544    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778546    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778549    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778551    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778554    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778557    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778560    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778562    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778565    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778568    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778570    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778573    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778576    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778578    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778581    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.781743 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778583    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778586    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778589    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778592    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778595    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778597    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778601    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778603    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778607    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778610    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778612    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778615    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778618    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778620    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778623    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.778625    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779209    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779215    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779218    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.782229 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779221    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779224    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779226    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779229    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779232    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779234    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779237    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779239    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779242    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779245    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779247    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779250    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779253    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779256    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779259    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779262    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779267    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779269    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779272    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779275    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.782701 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779277    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779293    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779297    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779299    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779303    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779305    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779308    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779311    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779313    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779317    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779320    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779322    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779325    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779328    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779330    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779333    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779335    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779338    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779341    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779343    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.783193 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779346    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779349    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779352    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779354    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779357    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779359    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779362    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779364    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779368    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779371    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779374    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779376    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779379    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779381    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779389    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779392    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779395    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779398    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779400    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779403    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.783719 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779405    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779408    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779411    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779414    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779416    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779419    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779427    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779430    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779432    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779435    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779437    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779442    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779445    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779448    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779452    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779456    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779459    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779461    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779464    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.784195 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779466    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779470    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779473    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779476    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779479    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779482    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779484    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779487    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779496    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779498    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779501    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779504    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779506    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779509    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779511    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779513    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779516    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779518    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779521    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.784666 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779524    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779526    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779529    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779531    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.779535    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779626    7019 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779640    7019 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779664    7019 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779669    7019 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779674    7019 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779677    7019 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779681    7019 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779685    7019 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779689    7019 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779694    7019 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779697    7019 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779701    7019 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779706    7019 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779709    7019 flags.go:64] FLAG: --cgroup-root=""
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779712    7019 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779716    7019 flags.go:64] FLAG: --client-ca-file=""
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779719    7019 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779723    7019 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 14:56:45.785221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779730    7019 flags.go:64] FLAG: --cluster-domain=""
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779733    7019 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779737    7019 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779740    7019 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779743    7019 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779747    7019 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779750    7019 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779753    7019 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779757    7019 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779760    7019 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779763    7019 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779766    7019 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779769    7019 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779772    7019 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779776    7019 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779780    7019 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779783    7019 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779786    7019 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779789    7019 flags.go:64] FLAG: --enable-server="true"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779792    7019 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779796    7019 flags.go:64] FLAG: --event-burst="100"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779800    7019 flags.go:64] FLAG: --event-qps="50"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779803    7019 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779808    7019 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 14:56:45.786139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779811    7019 flags.go:64] FLAG: --eviction-hard=""
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779815    7019 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779820    7019 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779823    7019 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779826    7019 flags.go:64] FLAG: --eviction-soft=""
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779830    7019 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779833    7019 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779836    7019 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779839    7019 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779842    7019 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779845    7019 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779848    7019 flags.go:64] FLAG: --feature-gates=""
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779852    7019 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779855    7019 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779859    7019 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779862    7019 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779865    7019 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779868    7019 flags.go:64] FLAG: --help="false"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779871    7019 flags.go:64] FLAG: --hostname-override="ip-10-0-129-45.ec2.internal"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779874    7019 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779877    7019 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779880    7019 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779884    7019 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 14:56:45.786895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779887    7019 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779891    7019 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779894    7019 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779896    7019 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779900    7019 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779903    7019 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779906    7019 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779909    7019 flags.go:64] FLAG: --kube-reserved=""
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779914    7019 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779917    7019 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779921    7019 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779925    7019 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779928    7019 flags.go:64] FLAG: --lock-file=""
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779930    7019 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779934    7019 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779937    7019 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779942    7019 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779945    7019 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779949    7019 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779952    7019 flags.go:64] FLAG: --logging-format="text"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779955    7019 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779958    7019 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779961    7019 flags.go:64] FLAG: --manifest-url=""
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779964    7019 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779969    7019 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 14:56:45.787447 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779972    7019 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779975    7019 flags.go:64] FLAG: --max-pods="110"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779978    7019 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779982    7019 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779985    7019 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779988    7019 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779991    7019 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779994    7019 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.779997    7019 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780004    7019 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780007    7019 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780010    7019 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780014    7019 flags.go:64] FLAG: --pod-cidr=""
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780016    7019 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780019    7019 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780024    7019 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780028    7019 flags.go:64] FLAG: --port="10250"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780031    7019 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780035    7019 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bb0bfb19f65ff53e"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780038    7019 flags.go:64] FLAG: --qos-reserved=""
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780041    7019 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780044    7019 flags.go:64] FLAG: --register-node="true"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780047    7019 flags.go:64] FLAG: --register-with-taints=""
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780051    7019 flags.go:64] FLAG: --registry-burst="10"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780054    7019 flags.go:64] FLAG: --registry-qps="5"
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780057    7019 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 14:56:45.788069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780061    7019 flags.go:64] FLAG: --reserved-memory=""
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780065    7019 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780068    7019 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780071    7019 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780074    7019 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780077    7019 flags.go:64] FLAG: --runonce="false"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780080    7019 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780083    7019 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780086    7019 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780089    7019 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780092    7019 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780095    7019 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780098    7019 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780101    7019 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780105    7019 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780107    7019 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780110    7019 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780113    7019 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780116    7019 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780120    7019 flags.go:64] FLAG: --system-cgroups=""
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780123    7019 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780129    7019 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780132    7019 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780135    7019 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780140    7019 flags.go:64] FLAG: --tls-min-version=""
Nov 02 14:56:45.788744 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780143    7019 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780146    7019 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780149    7019 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780153    7019 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780156    7019 flags.go:64] FLAG: --v="2"
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780166    7019 flags.go:64] FLAG: --version="false"
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780170    7019 flags.go:64] FLAG: --vmodule=""
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780174    7019 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780183    7019 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780379    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780383    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780387    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780390    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780394    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780397    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780400    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780402    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780405    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780408    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780411    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780414    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.789391 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780416    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780419    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780422    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780424    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780427    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780431    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780435    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780439    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780443    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780446    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780450    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780453    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780455    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780458    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780461    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780464    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780467    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780469    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780472    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.789910 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780474    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780484    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780487    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780490    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780493    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780495    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780498    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780501    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780504    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780506    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780509    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780512    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780514    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780517    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780520    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780522    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780525    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780528    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780530    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.790584 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780533    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780536    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780538    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780541    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780545    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780548    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780550    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780553    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780556    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780559    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780561    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780564    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780566    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780569    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780572    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780580    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780583    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780586    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780588    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.791350 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780591    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780594    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780596    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780599    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780601    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780604    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780607    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780611    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780615    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780618    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780621    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780624    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780627    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780629    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780632    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780635    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780637    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780642    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780656    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.791939 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780659    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780662    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780665    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780667    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780670    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780673    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780676    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780678    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780681    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780684    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780692    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780696    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780698    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780701    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780704    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780706    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780709    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.792540 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780712    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.780738    7019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780789    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780793    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780797    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780800    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780808    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780811    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780814    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780817    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780819    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780822    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780825    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780828    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780834    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.793311 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780837    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780840    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780843    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780846    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780848    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780851    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780854    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780856    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780859    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780862    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780864    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780874    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780877    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780880    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780883    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780885    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780888    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780891    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780893    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780896    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.794022 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780898    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780901    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780903    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780906    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780909    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780911    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780914    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780917    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780920    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780922    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780925    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780931    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780934    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780936    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780939    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780942    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780945    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780947    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780950    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780953    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.794610 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780956    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780958    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780961    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780964    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780974    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780978    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780982    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780985    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780987    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780990    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780993    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780995    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.780998    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781000    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781003    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781006    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781008    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781011    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781014    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.795316 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781016    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781019    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781021    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781024    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781028    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781031    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781033    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781037    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781040    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781042    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781045    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781048    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781050    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781053    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781056    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781058    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781063    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781065    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781073    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.795887 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781077    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781079    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781082    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781085    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781087    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781090    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781093    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781095    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781099    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781101    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781104    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781107    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781109    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781112    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.781114    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.785556    7019 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.785574    7019 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.785598    7019 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.785620    7019 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 14:56:45.796347 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785889    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785909    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785916    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785922    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785927    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785932    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785936    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785940    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785944    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785949    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785953    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785957    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785968    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785972    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785977    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785981    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785985    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785990    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785994    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.785998    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.796833 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786002    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786006    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786011    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786015    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786024    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786028    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786033    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786037    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786041    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786045    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786051    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786056    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786062    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786125    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786272    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786281    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786285    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786288    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786292    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.797306 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786295    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786299    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786302    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786305    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786308    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786311    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786314    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786317    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786320    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786323    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786326    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786329    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786332    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786335    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786337    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786340    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786343    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786346    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786349    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.797775 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786351    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786354    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786357    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786360    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786362    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786365    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786367    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786370    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786374    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786377    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786380    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786383    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786385    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786389    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786392    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786395    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786398    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786401    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786404    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786406    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.798312 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786409    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786412    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786414    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786417    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786419    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786422    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786425    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786428    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786431    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786433    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786436    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786439    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786442    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786444    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786447    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786450    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786452    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786458    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786461    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.798816 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786464    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786467    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786470    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786473    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786476    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786478    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786481    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786483    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786486    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.786521    7019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786703    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786709    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786712    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786716    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786719    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786722    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.799270 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786725    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786728    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786731    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786734    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786736    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786746    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786749    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786751    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786755    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786758    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786760    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786763    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786766    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786768    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786771    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786773    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786776    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786780    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786782    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786785    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.799721 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786788    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786790    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786793    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786795    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786798    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786801    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786803    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786806    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786809    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786812    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786814    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786817    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786823    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786828    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786831    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786834    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786837    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786841    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786843    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.800202 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786846    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786849    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786852    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786855    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786857    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786860    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786862    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786865    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786868    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786870    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786874    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786878    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786881    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786884    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786887    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786889    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786893    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786896    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786898    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786901    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.800676 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786904    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786907    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786910    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786913    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786915    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786918    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786927    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786930    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786933    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786935    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786939    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786942    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786945    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786947    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786950    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786953    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786955    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786958    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786961    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786964    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.801192 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786966    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786969    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786972    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786974    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786977    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786979    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786982    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786985    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786987    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786990    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786993    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786995    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.786998    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787000    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787003    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787005    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787009    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787013    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787017    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.801686 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787022    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787025    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.787051    7019 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787084    7019 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787089    7019 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787093    7019 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787097    7019 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787101    7019 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787104    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787107    7019 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787110    7019 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787112    7019 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787115    7019 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787118    7019 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787121    7019 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787124    7019 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 14:56:45.802196 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787127    7019 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787129    7019 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787132    7019 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787135    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787138    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787141    7019 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787143    7019 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787146    7019 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787148    7019 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787151    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787154    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787156    7019 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787159    7019 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787161    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787164    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787167    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787171    7019 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787174    7019 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787177    7019 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787181    7019 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 14:56:45.802579 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787184    7019 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787186    7019 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787189    7019 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787192    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787194    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787197    7019 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787199    7019 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787202    7019 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787205    7019 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787208    7019 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787210    7019 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787213    7019 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787216    7019 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787218    7019 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787221    7019 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787223    7019 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787228    7019 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787231    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787234    7019 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 14:56:45.803329 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787237    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787239    7019 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787242    7019 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787245    7019 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787247    7019 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787250    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787252    7019 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787255    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787258    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787262    7019 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787265    7019 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787268    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787272    7019 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787274    7019 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787277    7019 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787279    7019 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787291    7019 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787294    7019 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787296    7019 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787299    7019 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 14:56:45.804081 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787302    7019 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787304    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787307    7019 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787310    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787312    7019 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787316    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787318    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787321    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787324    7019 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787326    7019 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787329    7019 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787332    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787334    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787337    7019 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787339    7019 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787342    7019 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787345    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787348    7019 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787350    7019 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 14:56:45.804619 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787353    7019 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787356    7019 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787359    7019 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787362    7019 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787365    7019 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787370    7019 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787372    7019 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787375    7019 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787378    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787381    7019 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787384    7019 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787386    7019 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787389    7019 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787392    7019 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:45.787394    7019 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.787452    7019 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.788389    7019 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.788448    7019 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.788659    7019 server.go:1014] "Starting client certificate rotation"
Nov 02 14:56:45.805098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.788743    7019 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.788793    7019 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:50:34 +0000 UTC" deadline="2028-06-11 08:12:51.271672855 +0000 UTC"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.788826    7019 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14081h16m5.482848442s"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.789088    7019 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.790143    7019 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.791953    7019 log.go:25] "Validated CRI v1 runtime API"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.794359    7019 log.go:25] "Validated CRI v1 image API"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.795112    7019 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 14:56:45.805569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.797436    7019 fs.go:135] Filesystem UUIDs: map[4e612a45-5c89-46eb-abc7-a50bd3708726:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9e14c4db-ebfb-4212-a030-72a6c932702b:/dev/nvme0n1p3]
Nov 02 14:56:45.805988 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.797456    7019 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03563b5a37e36d6024dbafefac4a9d4a2e88b26070913d492e33be352098375d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03563b5a37e36d6024dbafefac4a9d4a2e88b26070913d492e33be352098375d/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09dbf9b29b2967d6df9539f0392e68195a5042b74c351ffa037ac7948da66fba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09dbf9b29b2967d6df9539f0392e68195a5042b74c351ffa037ac7948da66fba/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0bdfd6fce571361dd95f339233f628494ba762123efb20054033f5b8aecfd242/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0bdfd6fce571361dd95f339233f628494ba762123efb20054033f5b8aecfd242/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32bef338037d22e11dba9febf7f048f54e4ee84110c8620b305c99ca7c7f0501/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32bef338037d22e11dba9febf7f048f54e4ee84110c8620b305c99ca7c7f0501/userdata/shm major:0 minor:505 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d0ff24a47261e75c9e2bdcc70efcebdb25a58688b5d309bcb5572b46b432a9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d0ff24a47261e75c9e2bdcc70efcebdb25a58688b5d309bcb5572b46b432a9f/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3facf7fca9c04659b8c62219dcff6e59452a1817f042f3e4f6530ed03886ac26/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3facf7fca9c04659b8c62219dcff6e59452a1817f042f3e4f6530ed03886ac26/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fcd9979316bc5a76dd6faa25d7dfc0568e064c067ec99385d312b46b745e04f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fcd9979316bc5a76dd6faa25d7dfc0568e064c067ec99385d312b46b745e04f/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49e27abbc9f345773c85b7f2ddcd2adb929d522ffbd976558bfcc2fd0d7de287/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49e27abbc9f345773c85b7f2ddcd2adb929d522ffbd976558bfcc2fd0d7de287/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc/userdata/shm major:0 minor:444 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d7372812047d0ba0fb5c170b6ee62d0719e2a65f6477705b8b9647d3ba9a7cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d7372812047d0ba0fb5c170b6ee62d0719e2a65f6477705b8b9647d3ba9a7cf/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/937c3df4f33852f5d87707288b369b5eaf7f03fbff9c0993c12444275c8fbc18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/937c3df4f33852f5d87707288b369b5eaf7f03fbff9c0993c12444275c8fbc18/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94f7ce2e142253298f792915915ec82a3c36e6019171a57aa3672b45879a73fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94f7ce2e142253298f792915915ec82a3c36e6019171a57aa3672b45879a73fc/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3db888c9ffb7cc0201660ff7b957961dc033e25b5075dcc7e41ab27a52f6e73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3db888c9ffb7cc0201660ff7b957961dc033e25b5075dcc7e41ab27a52f6e73/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7743dc52b8c3cf6888e56afb3a50e50c5094c5bbacf8ecbb6a3e89b250ae0ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7743dc52b8c3cf6888e56afb3a50e50c5094c5bbacf8ecbb6a3e89b250ae0ea/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c685a0bde3520f72756305ed36f255dd468021b39de0470f942090a124b62bdf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c685a0bde3520f72756305ed36f255dd468021b39de0470f942090a124b62bdf/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2ef7f7f77387e2b61fa8a1bb4b397b2469cc2bf80f8c50e3a16675311135aab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2ef7f7f77387e2b61fa8a1bb4b397b2469cc2bf80f8c50e3a16675311135aab/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec960c0e56baa2c6e9dd89ae7ed8e1b3d02cc6af317a3055d6baebb5e1d94304/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec960c0e56baa2c6e9dd89ae7ed8e1b3d02cc6af317a3055d6baebb5e1d94304/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecaaaf25edd0c569d11c4cc78a243e0d83beb332dca49c00d5f1fe35c5d8ff21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecaaaf25edd0c569d11c4cc78a243e0d83beb332dca49c00d5f1fe35c5d8ff21/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1265bf239a70818a94c4f7b7196faed9734500524f5a4bb22c5549b845b5fbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1265bf239a70818a94c4f7b7196faed9734500524f5a4bb22c5549b845b5fbe/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb0b738f1485ce90d6752e9394772345c13ca0be9db2dd061e008189ddd89ae1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb0b738f1485ce90d6752e9394772345c13ca0be9db2dd061e008189ddd89ae1/userdata/shm major:0 minor:501 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdb804e389c36a44ed5d20fdd4cf6b6f6cc0f08279f6407758eeddd62a71768b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdb804e389c36a44ed5d20fdd4cf6b6f6cc0f08279f6407758eeddd62a71768b/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe7861a94ec7e5837a6b3685d9dda34fb14c06f334b490e720ce085022c6e98a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe7861a94ec7e5837a6b3685d9dda34fb14c06f334b490e720ce085022c6e98a/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff493e3e73dd9b01267f73c40b92683d80615c0994558b5b3df4e1af910f9f67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff493e3e73dd9b01267f73c40b92683d80615c0994558b5b3df4e1af910f9f67/userdata/shm major:0 minor:503 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/044f0c98-3b8f-4283-9bdf-d235a867e360/volumes/kubernetes.io~projected/kube-api-access-x7dtr:{mountpoint:/var/lib/kubelet/pods/044f0c98-3b8f-4283-9bdf-d235a867e360/volumes/kubernetes.io~projected/kube-api-access-x7dtr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10ef5542-79bd-4e48-866d-952fbeb2bad2/volumes/kubernetes.io~projected/kube-api-access-lxlz4:{mountpoint:/var/lib/kubelet/pods/10ef5542-79bd-4e48-866d-952fbeb2bad2/volumes/kubernetes.io~projected/kube-api-access-lxlz4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~projected/kube-api-access-tqc8c:{mountpoint:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~projected/kube-api-access-tqc8c major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/config-reloader/2 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/prometheus/15 major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~empty-dir/config-out major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~projected/kube-api-access-nfj5v:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~projected/kube-api-access-nfj5v major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~projected/tls-assets major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/config major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:417 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:418 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/web-config major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volumes/kubernetes.io~projected/kube-api-access-bhwm4:{mountpoint:/var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volumes/kubernetes.io~projected/kube-api-access-bhwm4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e1b988f-c2c6-4844-84ea-1146294c99b8/volumes/kubernetes.io~projected/kube-api-access-b7xjr:{mountpoint:/var/lib/kubelet/pods/2e1b988f-c2c6-4844-84ea-1146294c99b8/volumes/kubernetes.io~projected/kube-api-access-b7xjr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e1b988f-c2c6-4844-84ea-1146294c99b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/2e1b988f-c2c6-4844-84ea-1146294c99b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35254468-9d2e-4c69-a162-02ae0854b07f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/35254468-9d2e-4c69-a162-02ae0854b07f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:499 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3558d580-6e93-4b65-bc56-98234fa61562/volumes/kubernetes.io~projected/kube-api-access-c9q75:{mountpoint:/var/lib/kubelet/pods/3558d580-6e93-4b65-bc56-98234fa61562/volumes/kubernetes.io~projected/kube-api-access-c9q75 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~projected/kube-api-access-t7wt7:{mountpoint:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~projected/kube-api-access-t7wt7 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ab14eae-d5f0-46e3-abda-c817d0e8b66e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4ab14eae-d5f0-46e3-abda-c817d0e8b66e/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f/volumes/kubernetes.io~projected/kube-api-access-c5ppl:{mountpoint:/var/lib/kubelet/pods/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f/volumes/kubernetes.io~projected/kube-api-access-c5ppl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:495 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67bc4f60-d5c6-4390-a61e-7898d781b4c2/volumes/kubernetes.io~projected/kube-api-access-9s8kn:{mountpoint:/var/lib/kubelet/pods/67bc4f60-d5c6-4390-a61e-7898d781b4c2/volumes/kubernetes.io~projected/kube-api-access-9s8kn major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67bc4f60-d5c6-4390-a61e-7898d781b4c2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/67bc4f60-d5c6-4390-a61e-7898d781b4c2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~projected/kube-api-access-rtsf7:{mountpoint:/var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~projected/kube-api-access-rtsf7 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c1e0c29-c679-40f0-9961-7f7ff476e04e/volumes/kubernetes.io~projected/kube-api-access-n89t4:{mountpoint:/var/lib/kubelet/pods/7c1e0c29-c679-40f0-9961-7f7ff476e04e/volumes/kubernetes.io~projected/kube-api-access-n89t4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e7d86c9-eec3-46af-91eb-e1a2851a59d4/volumes/kubernetes.io~projected/kube-api-access-qnz9s:{mountpoint:/var/lib/kubelet/pods/8e7d86c9-eec3-46af-91eb-e1a2851a59d4/volumes/kubernetes.io~projected/kube-api-access-qnz9s major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~projected/kube-api-access-lq9g8:{mountpoint:/var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~projected/kube-api-access-lq9g8 major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93eca203-b6a7-41d2-b640-953dd6ccaa02/volumes/kubernetes.io~projected/kube-api-access-ggpqm:{mountpoint:/var/lib/kubelet/pods/93eca203-b6a7-41d2-b640-953dd6ccaa02/volumes/kubernetes.io~projected/kube-api-access-ggpqm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~projected/kube-api-access-2p5nf:{mountpoint:/var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~projected/kube-api-access-2p5nf major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~projected/kube-api-access-t4tjr:{mountpoint:/var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~projected/kube-api-access-t4tjr major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/alertmanager/9 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~empty-dir/config-out major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~projected/kube-api-access-h5b4q:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~projected/kube-api-access-h5b4q major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~projected/tls-assets major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/config-volume major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/web-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbfbe419-497a-489b-9109-c52ed3bf0a6e/volumes/kubernetes.io~projected/kube-api-access-ks6p6:{mountpoint:/var/lib/kubelet/pods/bbfbe419-497a-489b-9109-c52ed3bf0a6e/volumes/kubernetes.io~projected/kube-api-access-ks6p6 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbfbe419-497a-489b-9109-c52ed3bf0a6e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bbfbe419-497a-489b-9109-c52ed3bf0a6e/volumes/kubernetes.io~secret/cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be62e46e-7648-4e29-a2ba-fce4655087ee/volumes/kubernetes.io~projected/kube-api-access-nv26p:{mountpoint:/var/lib/kubelet/pods/be62e46e-7648-4e29-a2ba-fce4655087ee/volumes/kubernetes.io~projected/kube-api-access-nv26p major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be62e46e-7648-4e29-a2ba-fce4655087ee/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/be62e46e-7648-4e29-a2ba-fce4655087ee/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce469d24-f9cc-4381-be8a-5c9006c21db7/volumes/kubernetes.io~projected/kube-api-access-65ps9:{mountpoint:/var/lib/kubelet/pods/ce469d24-f9cc-4381-be8a-5c9006c21db7/volumes/kubernetes.io~projected/kube-api-access-65ps9 major:0 minor:500 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~projected/kube-api-access-dt5t4:{mountpoint:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~projected/kube-api-access-dt5t4 major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:324 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/777265bda03c8931cb45745d822451f6d4a935399775b7b5b1b0d630e458ed0c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ff94d52d64d1612702b75913d8cd9df111b59f74bd9e00166b11531255ab197a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/4d7b49d0d6b417cb5b1203c4b3bad612b05e652323ca9268444f1fb5c391e201/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/67b35c779fd645a1af589ce3d88bc5db1edd6d7515bf615895ebef99311867fd/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/abf290023a8f69fcc634720d6452acfef86791a4be9591d6cd9208ad7113fe79/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5e9e223e0215cd18c9aaf5bb5f0dd4cfb74325f186fde3e7dfdbcc21ae2f6629/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/608997c57f7a074628d7408f1d55f694cba4fcde71c4109c5753ab78dd898e86/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/03666045d13b0dfac07bd3c2e96cbf9d256356b226e88654c79798ff80e6210f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/00e97b84097140897bd44e2c4deada3eb5849da792e4be560d5a40857ae12892/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/366cf79e76d3ccd261401537eac7d4610a3893365350b38bdc2c41264be723d6/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/cf92439f34ea1ff3cabeafa2107f0d77cd67a8b0a98c5a5d2540c7d460506b4c/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/efca6fc8b71246d65eaf9cd977b6efc06847788c9a125b1021154873bc7926df/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/db93b0ed146a011d87d148324c1345915d52004838f33487a213c62df20f8dcb/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b9ec864d98fe6a447a808b25214a1c95e7062e6109812255073bde1c6deac8e5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4f8cc452fbb7c420e5036db70b64978dc4755f001613e258b7af245a6d843f45/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/78815b9c3d956f092d45f52bc60e8a62fae1c16ac438e15447fce93dd417ce4a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1d095dd6c6ed334a7b684359f3e18ae590638676396d98c73be57a91414e3132/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a372daa18a99f9f167cc9cd9754c061de487dc55e930d86e73f0384c21a19770/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/847c325e71745c60a250890f777029a99f6cc85e0e3d93f0a3e45c5538f9140f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/dae601234f0709511d0aac0252463f62f6c99cbcc8fe893d845b3b160e7c2276/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/0a066c7651c93bc43d5cde84cc55b983062ffc08a77c63b77c80f2c2eb76778a/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f2c427849c31c9f49fe0e96d88c9351dc8678568820d98a3d32b2108880ea7ad/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/dd02cac41fc27834297f0a6c29dcc10aa356eba4b027a4a59f4c7cbde3e7c4bb/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c019847110cc5fbe7c7f43292773806c9e7970cbe9530af882b85a2843089f76/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/e7af34eb20d5db82910de506e1342276b05b18341605d95671ecd439bfc91855/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/52c6971359092758202b32e4082757661811494313781e7938be647a583a1cae/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/206394de46015c2fe2d8f1f8b80fd39e0967931691059f23be53eed4b1e324da/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/d64cd9c3fe2cac1df1e52de5d1c90de3e56fcf0c6b0e84bf48cbbec5a70a5b1a/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/afe513dde6160c08df62a512f1dbd4a99c5c900fded4165a52e1ededac37470b/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d6079a10980969085d11eaebe87dc60bd87d35d167d7e7bfbd50d8ed305a17c4/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/4da608a99b78450e45495ca914ff0a4cb39dd71f176dc2db4975052238e74975/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/17c37f4500c511674bd421e424447503628c6d7abe81175cc330cb0758af7b85/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/1def0f098f7dedff0c762bd12dac701201f6258f797f2b9c89df453b29148663/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/6b00a9a00fcf512b06df269552b4cf7d445554784cf13d89d311de98ee6369a4/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/5fdb50adc2ba6a24b3e94123b14c3cbfd7cf36ad63f226a1353e0c524edb1335/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/026c688dc2b2b177695054c238b95b98c88498156936ff6cad228f3148a421ad/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/0d71bb636b0c382dfa1c180622b0a4a366392cba2b959df17703ebf6c04670e8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/30673a129bef7a326d12422b5989b82c0b00d4c01ed29d85878026e284df11be/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a9aecb1b5976c1d0c2a0809da9181f2ed03c3117c1b3582c2cc7c721c4f4587f/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/886f83cb6a67c6cda2903f91ee2c110aaddb731afd7e9083eb12649961cf3b79/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/deb1ebb73606fccafffcbdff5ef62786d215b9027c90e6aec1f291ebb11595a8/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/da0f16b548477387998bc9904a094d7a29a2afaf1ab5a92c23098fb72407eb4b/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/12308ef5e7377f48205b9ee3938918edf3bcd53740bd13208c6f5ab68a93b2fe/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/4f1e751f22d3fbac6c6f209b32ae34cb517323e8ce5d4395a5f2dbfbae602f1b/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/f9cfa759b2faca508c820e234303847d23fc99f6764e9a4b53df499bf061cd20/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/23295b2946923956307fe5593437ebe556e123eb88872467d3cfc7f71bebf910/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/af1727af41085bb7a2224989e08865dea46f9df8763cc242eda2261f6aea1d35/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/00ae95181dc61f8a11e5326f2baac01abc8311183304a331f757d39bd7e98ecf/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/c17e85351721e583baf7f59a112d0a56104f14ecad2c168950266784b1cac3da/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/d928129c1e994774bc058119a16ce3c86ed785869fb6918a132afec810d22ccc/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/9a49d4b1fa6667ac09092c82d954f05908b295f70337881ce1276975ca1442ba/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/7876625a61443fcda3249f81cc09eb6ffa60de4bdc56faaadf488222dba9adff/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/2eae9e0f3a6ca2f587b36e3ff9e9b6b52da984d226815b05344756548595b57c/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/4abfbe0e54479dda9b31247a192d6bbb5174acb704b6d93f58b9551635be9c80/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/00663d4246824cbf59e65f081e1443e98442a066d96a23dc29c6a757a14db57e/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/75a416e34e2f36a98b0d21e21522b10fe98aa21396f7b65df7e635275878eb96/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/a5877b643074711081410da593d87cec9fff568a6049cee565dad1e87db05fd8/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/d856510f6b5444f7e461524268a24743b40bbf3cbf3aa39675c05f8e4de79559/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-439:{mountpoint:/var/lib/containers/storage/overlay/703fdde889c18cb90552fe0f8ea9f816224161ece1215225ee5b79036064288f/merged major:0 minor:439 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/b0eb852b4fbd2182b12151be8bd9dd01c40da6bc8ac4c0da65132ba69bb0b04b/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/44f2d2fdbfd91baa6d692e518362b6ae3330d144ca2ddbe80620ae8a0690d8c6/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/15ab17059d95102babb5d367e30b0e83e56efa5c3d441d073ee2e0de5c4e0b22/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/59da1f38f824cd228b25afaac1855d88f4e9facc74a7c3989cf45042d6df3b3b/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/748d7e3e72f9f79ad37f497a05ded334eb1b1fb19ce8b589724f137fec18aecc/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/e3bf2c794c6d7ae1de9b9f428d46ef5be1e540e8714eef9ddbe807c057aaccee/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/0f1e5e2688d758dd9694ed3d0bedaf1a5ae1480e66ad8a166674ea8b76f59aac/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/2151197eb7b24e3acaa5023bccfbcc1d742289cc4b5ea46e1b66bb6fb0f2247f/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/2c5a3279de029bf8a186fa0ee22836559e119b01e5ce03e4660341983ad00ef2/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/fd7c9785847450b5e23d03f5ee9cf383432783fc29589b1c1ed0990753da6652/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-507:{mountpoint:/var/lib/containers/storage/overlay/a4f75eda8dd3504640d60f2d0d12ff430663ed193a26e224c52c7ffbb7ef0c4d/merged major:0 minor:507 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/3259ac2cf451a1a2ed547b74a5e200dfa7981dec71340a9f1a0ac9d4822279f4/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c19ae443499bbab1a160f8d9dc1b38f58a1b5474fe48fad005a4bf708e6c1265/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-511:{mountpoint:/var/lib/containers/storage/overlay/595dd5e6dfd135b5765ceec96156ff4cb276d881b5a837f885d2b93f91316012/merged major:0 minor:511 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/0038729b275b51aec6d92bd912c950969a9748e002b5053a512e43d786188413/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/08bfbc43cf28f78b979dc7c03b8f6fb57d05dc9d19422d29e20974d6bb1fa51c/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/537945d32f6da118575adb58a0e800cbbeb85017a059e858ff502f0e0c8c22aa/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/87b6b5d1d12cf169c063f09e42fabd515f55f60c23f8b08075e8fdb914b44e97/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/31a9b9f565e9339ca143ac8238407f6909907a150622f509ca018a59661d6023/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/306d25581064d8d9044d9d2db8679efdc98dea891301f21be1036536dc26edca/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a9de24c0ccdba7b1b886c56976be471d39c1069cbc9eff973dbb0aabec684690/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/aeeec9c1b126c275ab954ac1af0d4306d48a16c9947d67854d24b944b544468d/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c71f224ebd30f4cd12dc4de46a2268ab9ebe1fa38157d73517513364c64bf182/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/981e561b30a55e5d48fa1bd4bdfcb557498afe86d2765d7c72301de35c8c1e96/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6c1e1b1cddb68084d1fe3401edaac002f6f9e4c5846e837d5f87e4360fd591cc/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/436bc80c132fc3bab82c0db989d1ca80c5a11bb5ee3b07bc3b4ca1ca5ea3665e/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/7478ee7d10bddd1073f971dccc90930da8bc3a587d82895fd29874263eaf9c99/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/918f7c77505512871d8240afd23617aa993f8be5d9fa6d003765a6b5e41b6979/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 02 14:56:45.806491 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.806044    7019 manager.go:217] Machine: {Timestamp:2026-11-02 14:56:45.805577507 +0000 UTC m=+0.059531122 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108662 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f2069002d9bc5b7e024f8336c08d SystemUUID:ec22f206-9002-d9bc-5b7e-024f8336c08d BootID:b414a3d0-3856-4ff4-9150-cf1f013c30b3 Filesystems:[{Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc/userdata/shm DeviceMajor:0 DeviceMinor:444 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1265bf239a70818a94c4f7b7196faed9734500524f5a4bb22c5549b845b5fbe/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:320 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2ef7f7f77387e2b61fa8a1bb4b397b2469cc2bf80f8c50e3a16675311135aab/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3facf7fca9c04659b8c62219dcff6e59452a1817f042f3e4f6530ed03886ac26/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35254468-9d2e-4c69-a162-02ae0854b07f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:499 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c1e0c29-c679-40f0-9961-7f7ff476e04e/volumes/kubernetes.io~projected/kube-api-access-n89t4 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~projected/kube-api-access-2p5nf DeviceMajor:0 DeviceMinor:244 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdb804e389c36a44ed5d20fdd4cf6b6f6cc0f08279f6407758eeddd62a71768b/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be62e46e-7648-4e29-a2ba-fce4655087ee/volumes/kubernetes.io~projected/kube-api-access-nv26p DeviceMajor:0 DeviceMinor:362 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce469d24-f9cc-4381-be8a-5c9006c21db7/volumes/kubernetes.io~projected/kube-api-access-65ps9 DeviceMajor:0 DeviceMinor:500 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f/volumes/kubernetes.io~projected/kube-api-access-c5ppl DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03563b5a37e36d6024dbafefac4a9d4a2e88b26070913d492e33be352098375d/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:425 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-511 DeviceMajor:0 DeviceMinor:511 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecaaaf25edd0c569d11c4cc78a243e0d83beb332dca49c00d5f1fe35c5d8ff21/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:287 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67bc4f60-d5c6-4390-a61e-7898d781b4c2/volumes/kubernetes.io~projected/kube-api-access-9s8kn DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~projected/kube-api-access-h5b4q DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fcd9979316bc5a76dd6faa25d7dfc0568e064c067ec99385d312b46b745e04f/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bbfbe419-497a-489b-9109-c52ed3bf0a6e/volumes/kubernetes.io~projected/kube-api-access-ks6p6 DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~projected/kube-api-access-tqc8c DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/93eca203-b6a7-41d2-b640-953dd6ccaa02/volumes/kubernetes.io~projected/kube-api-access-ggpqm DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe7861a94ec7e5837a6b3685d9dda34fb14c06f334b490e720ce085022c6e98a/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:422 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:427 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d7372812047d0ba0fb5c170b6ee62d0719e2a65f6477705b8b9647d3ba9a7cf/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~projected/kube-api-access-nfj5v DeviceMajor:0 DeviceMinor:432 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb0b738f1485ce90d6752e9394772345c13ca0be9db2dd061e008189ddd89ae1/userdata/shm DeviceMajor:0 DeviceMinor:501 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:369 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/937c3df4f33852f5d87707288b369b5eaf7f03fbff9c0993c12444275c8fbc18/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~projected/kube-api-access-t4tjr DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:424 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbfbe419-497a-489b-9109-c52ed3bf0a6e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:285 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09dbf9b29b2967d6df9539f0392e68195a5042b74c351ffa037ac7948da66fba/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49e27abbc9f345773c85b7f2ddcd2adb929d522ffbd976558bfcc2fd0d7de287/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e1b988f-c2c6-4844-84ea-1146294c99b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0bdfd6fce571361dd95f339233f628494ba762123efb20054033f5b8aecfd242/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-439 DeviceMajor:0 DeviceMinor:439 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff493e3e73dd9b01267f73c40b92683d80615c0994558b5b3df4e1af910f9f67/userdata/shm DeviceMajor:0 DeviceMinor:503 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3558d580-6e93-4b65-bc56-98234fa61562/volumes/kubernetes.io~projected/kube-api-access-c9q75 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e7d86c9-eec3-46af-91eb-e1a2851a59d4/volumes/kubernetes.io~projected/kube-api-access-qnz9s DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32bef338037d22e11dba9febf7f048f54e4ee84110c8620b305c99ca7c7f0501/userdata/shm DeviceMajor:0 DeviceMinor:505 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94f7ce2e142253298f792915915ec82a3c36e6019171a57aa3672b45879a73fc/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/992dc344-ae2d-43e9-89df-42011f2e8794/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:326 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec960c0e56baa2c6e9dd89ae7ed8e1b3d02cc6af317a3055d6baebb5e1d94304/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:417 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67bc4f60-d5c6-4390-a61e-7898d781b4c2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:421 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-507 DeviceMajor:0 DeviceMinor:507 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~projected/kube-api-access-rtsf7 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a/volumes/kubernetes.io~projected/kube-api-access-dt5t4 DeviceMajor:0 DeviceMinor:329 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d0ff24a47261e75c9e2bdcc70efcebdb25a58688b5d309bcb5572b46b432a9f/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8ea0a743-d6e8-4211-bd0f-b719dfc91508/volumes/kubernetes.io~projected/kube-api-access-lq9g8 DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:279 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:428 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:495 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77b573dd-a446-4605-8362-2fd3e4565d3d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3db888c9ffb7cc0201660ff7b957961dc033e25b5075dcc7e41ab27a52f6e73/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:419 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/be62e46e-7648-4e29-a2ba-fce4655087ee/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:357 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:420 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volumes/kubernetes.io~projected/kube-api-access-bhwm4 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b56d86c-0e63-4e77-8fa7-469ebc91e866/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e1b988f-c2c6-4844-84ea-1146294c99b8/volumes/kubernetes.io~projected/kube-api-access-b7xjr DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ab14eae-d5f0-46e3-abda-c817d0e8b66e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/044f0c98-3b8f-4283-9bdf-d235a867e360/volumes/kubernetes.io~projected/kube-api-access-x7dtr DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:286 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d3e864f-07ce-47ff-a6f5-e4197c15f72d/volumes/kubernetes.io~projected/kube-api-access-t7wt7 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11e1276c-fe6b-44e7-8df9-a7925a8cd350/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:368 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:418 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:423 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10ef5542-79bd-4e48-866d-952fbeb2bad2/volumes/kubernetes.io~projected/kube-api-access-lxlz4 DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c685a0bde3520f72756305ed36f255dd468021b39de0470f942090a124b62bdf/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7743dc52b8c3cf6888e56afb3a50e50c5094c5bbacf8ecbb6a3e89b250ae0ea/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03563b5a37e36d6 MacAddress:4e:74:c3:6c:d2:dc Speed:10000 Mtu:8901} {Name:09dbf9b29b2967d MacAddress:b2:41:a1:4f:63:4b Speed:10000 Mtu:8901} {Name:0bdfd6fce571361 MacAddress:52:7a:fa:b4:85:07 Speed:10000 Mtu:8901} {Name:32bef338037d22e MacAddress:ae:a8:27:d5:24:52 Speed:10000 Mtu:8901} {Name:3d0ff24a47261e7 MacAddress:1a:ef:f7:64:26:0b Speed:10000 Mtu:8901} {Name:3facf7fca9c0465 MacAddress:62:11:3b:b8:db:96 Speed:10000 Mtu:8901} {Name:6fe1f2d0014a121 MacAddress:3a:cc:5e:3b:ca:42 Speed:10000 Mtu:8901} {Name:8d7372812047d0b MacAddress:72:10:57:13:81:20 Speed:10000 Mtu:8901} {Name:b3db888c9ffb7cc MacAddress:ee:55:43:60:b9:e4 Speed:10000 Mtu:8901} {Name:b7743dc52b8c3cf MacAddress:7e:fe:bf:3e:f0:c3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c0:44:f9:d9 Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:1a:0e:62:96:c0 Speed:0 Mtu:8901} {Name:ec960c0e56baa2c MacAddress:36:37:8e:77:66:ef Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c0:44:f9:d9 Speed:0 Mtu:9001} {Name:fb0b738f1485ce9 MacAddress:de:46:48:ed:ef:0d Speed:10000 Mtu:8901} {Name:fdb804e389c36a4 MacAddress:4e:ee:80:76:d6:0a Speed:10000 Mtu:8901} {Name:ff493e3e73dd9b0 MacAddress:1e:d0:9c:41:ab:80 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:5e:23:ab:a0:c7:18 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:92:d9:bc:67:2c:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 14:56:45.806758 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.806496    7019 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 14:56:45.806758 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.806540    7019 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 14:56:45.806871 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.806849    7019 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 14:56:45.807023 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.806873    7019 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 14:56:45.807063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807025    7019 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 14:56:45.807063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807033    7019 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 14:56:45.807063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807043    7019 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:56:45.807063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807056    7019 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 14:56:45.807169 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807132    7019 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 14:56:45.807295 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807279    7019 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 14:56:45.807407 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807398    7019 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 14:56:45.807444 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807435    7019 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 14:56:45.807473 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807444    7019 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 14:56:45.807473 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807454    7019 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 14:56:45.807473 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807462    7019 kubelet.go:405] "Adding apiserver pod source"
Nov 02 14:56:45.807473 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.807470    7019 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 14:56:45.808171 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.808156    7019 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:56:45.808171 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.808168    7019 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:56:45.808663 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.808629    7019 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 14:56:45.811850 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.811832    7019 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 14:56:45.811850 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.811846    7019 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 14:56:45.811966 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.811936    7019 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 14:56:45.812130 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812118    7019 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 14:56:45.812160 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812148    7019 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 14:56:45.812276 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812263    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 14:56:45.812316 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812283    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 14:56:45.812316 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812311    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812319    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812328    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812335    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812341    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812347    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812353    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812359    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 14:56:45.812372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812371    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 14:56:45.812618 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812380    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 14:56:45.812618 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812404    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 14:56:45.812618 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.812409    7019 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 14:56:45.815774 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.815757    7019 server.go:1263] "Started kubelet"
Nov 02 14:56:45.815981 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.815910    7019 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 14:56:45.816161 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.816146    7019 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 14:56:45.816218 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.815991    7019 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 14:56:45.816616 ip-10-0-129-45 systemd[1]: Started Kubernetes Kubelet.
Nov 02 14:56:45.816818 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.816801    7019 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 14:56:45.822225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.822205    7019 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.823132    7019 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.823153    7019 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.823307    7019 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:50:36 +0000 UTC" deadline="2028-08-03 22:06:44.881901114 +0000 UTC"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.823333    7019 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15367h9m59.058571375s"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.823356    7019 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.823372    7019 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:45.823384    7019 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-45.ec2.internal\" not found"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.823465    7019 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.825332    7019 factory.go:55] Registering systemd factory
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.825353    7019 factory.go:223] Registration of the systemd container factory successfully
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.825576    7019 factory.go:153] Registering CRI-O factory
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.825592    7019 factory.go:223] Registration of the crio container factory successfully
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.825635    7019 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.825669    7019 factory.go:103] Registering Raw factory
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.825695    7019 manager.go:1199] Started watching for new ooms in manager
Nov 02 14:56:45.826704 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.826050    7019 manager.go:319] Starting recovery of all containers
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.826947    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f" volumeName="kubernetes.io/projected/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-kube-api-access-c5ppl" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.826982    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-client-tls" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827000    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827014    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d3e864f-07ce-47ff-a6f5-e4197c15f72d" volumeName="kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-tls" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827027    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" volumeName="kubernetes.io/empty-dir/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-textfile" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827055    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827086    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" volumeName="kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827119    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be62e46e-7648-4e29-a2ba-fce4655087ee" volumeName="kubernetes.io/projected/be62e46e-7648-4e29-a2ba-fce4655087ee-kube-api-access-nv26p" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827137    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/configmap/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827150    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827173    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827187    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827201    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1b988f-c2c6-4844-84ea-1146294c99b8" volumeName="kubernetes.io/secret/2e1b988f-c2c6-4844-84ea-1146294c99b8-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827263    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" volumeName="kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827292    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827306    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827334    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827347    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827359    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827381    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827395    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67bc4f60-d5c6-4390-a61e-7898d781b4c2" volumeName="kubernetes.io/projected/67bc4f60-d5c6-4390-a61e-7898d781b4c2-kube-api-access-9s8kn" seLinuxMountContext=""
Nov 02 14:56:45.827457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827407    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1b988f-c2c6-4844-84ea-1146294c99b8" volumeName="kubernetes.io/empty-dir/2e1b988f-c2c6-4844-84ea-1146294c99b8-data-volume" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827435    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d3e864f-07ce-47ff-a6f5-e4197c15f72d" volumeName="kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827514    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" volumeName="kubernetes.io/projected/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-kube-api-access-t4tjr" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827525    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827576    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eca203-b6a7-41d2-b640-953dd6ccaa02" volumeName="kubernetes.io/projected/93eca203-b6a7-41d2-b640-953dd6ccaa02-kube-api-access-ggpqm" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827591    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827604    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce469d24-f9cc-4381-be8a-5c9006c21db7" volumeName="kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827639    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/empty-dir/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-tmp" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827667    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1b988f-c2c6-4844-84ea-1146294c99b8" volumeName="kubernetes.io/configmap/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827679    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" volumeName="kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827705    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" volumeName="kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827733    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827745    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d3e864f-07ce-47ff-a6f5-e4197c15f72d" volumeName="kubernetes.io/empty-dir/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-audit-log" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827772    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" volumeName="kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827788    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea0a743-d6e8-4211-bd0f-b719dfc91508" volumeName="kubernetes.io/empty-dir/8ea0a743-d6e8-4211-bd0f-b719dfc91508-volume-directive-shadow" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827828    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827854    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827867    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827878    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827920    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="044f0c98-3b8f-4283-9bdf-d235a867e360" volumeName="kubernetes.io/projected/044f0c98-3b8f-4283-9bdf-d235a867e360-kube-api-access-x7dtr" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827935    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.827974    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67bc4f60-d5c6-4390-a61e-7898d781b4c2" volumeName="kubernetes.io/empty-dir/67bc4f60-d5c6-4390-a61e-7898d781b4c2-tmp-dir" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828001    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3558d580-6e93-4b65-bc56-98234fa61562" volumeName="kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-cni-binary-copy" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828013    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992dc344-ae2d-43e9-89df-42011f2e8794" volumeName="kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828024    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828100    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67bc4f60-d5c6-4390-a61e-7898d781b4c2" volumeName="kubernetes.io/secret/67bc4f60-d5c6-4390-a61e-7898d781b4c2-metrics-tls" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828117    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828139    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828162    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828173    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828184    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b56d86c-0e63-4e77-8fa7-469ebc91e866" volumeName="kubernetes.io/secret/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828212    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828253    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828271    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c1e0c29-c679-40f0-9961-7f7ff476e04e" volumeName="kubernetes.io/projected/7c1e0c29-c679-40f0-9961-7f7ff476e04e-kube-api-access-n89t4" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828305    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea0a743-d6e8-4211-bd0f-b719dfc91508" volumeName="kubernetes.io/projected/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-api-access-lq9g8" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828346    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10ef5542-79bd-4e48-866d-952fbeb2bad2" volumeName="kubernetes.io/projected/10ef5542-79bd-4e48-866d-952fbeb2bad2-kube-api-access-lxlz4" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828358    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828383    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b573dd-a446-4605-8362-2fd3e4565d3d" volumeName="kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-tmp" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828398    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828428    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 02 14:56:45.828448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828449    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828479    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d3e864f-07ce-47ff-a6f5-e4197c15f72d" volumeName="kubernetes.io/projected/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-kube-api-access-t7wt7" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828490    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b573dd-a446-4605-8362-2fd3e4565d3d" volumeName="kubernetes.io/projected/77b573dd-a446-4605-8362-2fd3e4565d3d-kube-api-access-rtsf7" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828528    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c1e0c29-c679-40f0-9961-7f7ff476e04e" volumeName="kubernetes.io/configmap/7c1e0c29-c679-40f0-9961-7f7ff476e04e-serviceca" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828539    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828551    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/projected/11e1276c-fe6b-44e7-8df9-a7925a8cd350-kube-api-access-tqc8c" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828599    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d3e864f-07ce-47ff-a6f5-e4197c15f72d" volumeName="kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-client-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828613    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea0a743-d6e8-4211-bd0f-b719dfc91508" volumeName="kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828624    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992dc344-ae2d-43e9-89df-42011f2e8794" volumeName="kubernetes.io/projected/992dc344-ae2d-43e9-89df-42011f2e8794-kube-api-access-2p5nf" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828686    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b56d86c-0e63-4e77-8fa7-469ebc91e866" volumeName="kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-env-overrides" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828702    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d3e864f-07ce-47ff-a6f5-e4197c15f72d" volumeName="kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828727    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93eca203-b6a7-41d2-b640-953dd6ccaa02" volumeName="kubernetes.io/empty-dir/93eca203-b6a7-41d2-b640-953dd6ccaa02-tmp-dir" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828751    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d3e864f-07ce-47ff-a6f5-e4197c15f72d" volumeName="kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828766    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea0a743-d6e8-4211-bd0f-b719dfc91508" volumeName="kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828777    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be62e46e-7648-4e29-a2ba-fce4655087ee" volumeName="kubernetes.io/secret/be62e46e-7648-4e29-a2ba-fce4655087ee-monitoring-plugin-cert" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828803    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-grpc-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828814    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35254468-9d2e-4c69-a162-02ae0854b07f" volumeName="kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828845    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10ef5542-79bd-4e48-866d-952fbeb2bad2" volumeName="kubernetes.io/configmap/10ef5542-79bd-4e48-866d-952fbeb2bad2-iptables-alerter-script" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828889    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828911    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3558d580-6e93-4b65-bc56-98234fa61562" volumeName="kubernetes.io/projected/3558d580-6e93-4b65-bc56-98234fa61562-kube-api-access-c9q75" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828926    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828939    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992dc344-ae2d-43e9-89df-42011f2e8794" volumeName="kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828949    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828960    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab14eae-d5f0-46e3-abda-c817d0e8b66e" volumeName="kubernetes.io/secret/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-agent-certs" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828971    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828981    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e1b988f-c2c6-4844-84ea-1146294c99b8" volumeName="kubernetes.io/projected/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-api-access-b7xjr" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.828994    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" volumeName="kubernetes.io/projected/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-kube-api-access-qnz9s" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829006    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829018    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b56d86c-0e63-4e77-8fa7-469ebc91e866" volumeName="kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829030    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b56d86c-0e63-4e77-8fa7-469ebc91e866" volumeName="kubernetes.io/projected/1b56d86c-0e63-4e77-8fa7-469ebc91e866-kube-api-access-bhwm4" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829041    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77b573dd-a446-4605-8362-2fd3e4565d3d" volumeName="kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-tuned" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829063    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea0a743-d6e8-4211-bd0f-b719dfc91508" volumeName="kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829074    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="992dc344-ae2d-43e9-89df-42011f2e8794" volumeName="kubernetes.io/configmap/992dc344-ae2d-43e9-89df-42011f2e8794-metrics-client-ca" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829088    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfbe419-497a-489b-9109-c52ed3bf0a6e" volumeName="kubernetes.io/projected/bbfbe419-497a-489b-9109-c52ed3bf0a6e-kube-api-access-ks6p6" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829100    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfbe419-497a-489b-9109-c52ed3bf0a6e" volumeName="kubernetes.io/secret/bbfbe419-497a-489b-9109-c52ed3bf0a6e-cert" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829112    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f1313c9-5c99-44a9-914f-3bdbcfd9a23f" volumeName="kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829123    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67bc4f60-d5c6-4390-a61e-7898d781b4c2" volumeName="kubernetes.io/configmap/67bc4f60-d5c6-4390-a61e-7898d781b4c2-config-volume" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829135    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829145    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11e1276c-fe6b-44e7-8df9-a7925a8cd350" volumeName="kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-federate-client-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829158    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3558d580-6e93-4b65-bc56-98234fa61562" volumeName="kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-multus-daemon-config" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829170    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ab14eae-d5f0-46e3-abda-c817d0e8b66e" volumeName="kubernetes.io/configmap/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-konnectivity-ca" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829181    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/projected/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-kube-api-access-dt5t4" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829192    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" volumeName="kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829218    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ea0a743-d6e8-4211-bd0f-b719dfc91508" volumeName="kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829230    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" volumeName="kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829243    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829254    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829265    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829278    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b56d86c-0e63-4e77-8fa7-469ebc91e866" volumeName="kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-config" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829290    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" volumeName="kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829302    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cfe9b9-6abc-494a-aa25-201041f17bd1" volumeName="kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829333    7019 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e34c8a6c-4225-4d6f-9886-9f1f908d2f5a" volumeName="kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829346    7019 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 14:56:45.830328 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.829353    7019 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 14:56:45.852120 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.852078    7019 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 14:56:45.853792 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.853762    7019 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 14:56:45.853792 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.853795    7019 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 14:56:45.853956 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.853821    7019 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 14:56:45.853956 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:45.853863    7019 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 14:56:45.870435 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.870410    7019 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 14:56:45.874260 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.874233    7019 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7" exitCode=0
Nov 02 14:56:45.879266 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.879238    7019 generic.go:356] "Generic (PLEG): container finished" podID="a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3" containerID="2290f6905a8abf97609fa8b8538552a503cb907b11af86286f65547215060b38" exitCode=0
Nov 02 14:56:45.881448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.881424    7019 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:45.883018 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.882999    7019 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:45.883689 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.883668    7019 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:45.892441 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.892418    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675" exitCode=0
Nov 02 14:56:45.895257 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.895213    7019 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 14:56:45.906169 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.906148    7019 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="6a7d158955cc8706d1ca90bf9e20f0677a56dd6cc3b840f38c19268d4f63719d" exitCode=0
Nov 02 14:56:45.906169 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.906170    7019 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="1c212160d005aaf5e1653879fe3e829e4eb291c840452dda1933c1d7647b9419" exitCode=0
Nov 02 14:56:45.906271 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.906181    7019 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="2837e0eff8b3a7178c643708d5c8316506232522d49aefab2f004b84a99a5996" exitCode=0
Nov 02 14:56:45.906271 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.906187    7019 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="f14323aad401518ec466327867954446d8af199699960d76ceeb7b00c3b4af82" exitCode=0
Nov 02 14:56:45.906271 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.906192    7019 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="5d959f778ba0ca8bb477b0fa3285b46377816892e7d21eea2f0b50f44362d566" exitCode=0
Nov 02 14:56:45.906271 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.906197    7019 generic.go:356] "Generic (PLEG): container finished" podID="8e7d86c9-eec3-46af-91eb-e1a2851a59d4" containerID="ce08091ec56539fcbc9edef4581568bb63a859095b81a56c1bdf4e7b12e10d56" exitCode=0
Nov 02 14:56:45.916899 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.916878    7019 generic.go:356] "Generic (PLEG): container finished" podID="2dacd25e67dd69f0add8a7632a4c6959" containerID="06baa112e4e66e7e61137707c5e44136be80145eb042faa388eab187c44556ad" exitCode=0
Nov 02 14:56:45.951566 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.951511    7019 manager.go:324] Recovery completed
Nov 02 14:56:45.953958 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:45.953936    7019 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 14:56:45.977791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977764    7019 cpu_manager.go:226] "Starting" policy="none"
Nov 02 14:56:45.977791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977783    7019 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 14:56:45.977919 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977802    7019 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 14:56:45.977949 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977921    7019 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 14:56:45.977949 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977932    7019 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 14:56:45.977949 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977948    7019 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 14:56:45.978075 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977953    7019 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 14:56:45.978075 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977959    7019 policy_none.go:50] "Start"
Nov 02 14:56:45.978075 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977968    7019 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 14:56:45.978075 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.977976    7019 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 14:56:45.979136 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.979122    7019 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 14:56:45.979172 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.979136    7019 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 14:56:45.979172 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.979142    7019 policy_none.go:44] "Start"
Nov 02 14:56:45.983921 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.983896    7019 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 14:56:45.986451 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.986427    7019 manager.go:341] "Starting Device Plugin manager"
Nov 02 14:56:45.986911 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:45.986457    7019 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 14:56:45.986911 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.986911    7019 server.go:87] "Starting device plugin registration server"
Nov 02 14:56:45.987165 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.987155    7019 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 14:56:45.987201 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.987166    7019 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 14:56:45.987277 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.987257    7019 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 14:56:45.987388 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.987357    7019 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 14:56:45.987460 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.987402    7019 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 14:56:45.987460 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.987410    7019 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 14:56:45.987556 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.987471    7019 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:56:45.98738725Z","UUID":"3e32dd15-c9a1-4353-b423-a661a03b0544","Handler":null,"Name":"","Endpoint":""}
Nov 02 14:56:45.989203 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.989171    7019 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 14:56:45.989290 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:45.989239    7019 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 14:56:45.989290 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:45.989177    7019 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 14:56:46.087950 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.087922    7019 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 14:56:46.098136 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.098118    7019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 14:56:46.098230 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.098146    7019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 14:56:46.098230 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.098155    7019 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-45.ec2.internal" event="NodeHasSufficientPID"
Nov 02 14:56:46.098230 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.098203    7019 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.138740 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.138715    7019 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.138847 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.138785    7019 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.155025 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.154982    7019 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"]
Nov 02 14:56:46.155708 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.155670    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"cf4d75b8d1582cef207a6bcc13a46c78","Type":"ContainerStarted","Data":"56bfbd8971a55248aa6511ebb93f2c8a88c50a4013813b14238f7ccc0ff6805a"}
Nov 02 14:56:46.155804 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.155720    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal" event={"ID":"cf4d75b8d1582cef207a6bcc13a46c78","Type":"ContainerStarted","Data":"091833c998736d87e0c5f5b22fa4d6505e0fafc4be5b331f7dd5c1483e2d0600"}
Nov 02 14:56:46.155804 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.155740    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"2dacd25e67dd69f0add8a7632a4c6959","Type":"ContainerStarted","Data":"31a45f2899ce79bac5a6683e3a8142d7d7d6a77896326af651c7f96737b91f97"}
Nov 02 14:56:46.155804 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.155750    7019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.155804 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.155758    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"2dacd25e67dd69f0add8a7632a4c6959","Type":"ContainerDied","Data":"06baa112e4e66e7e61137707c5e44136be80145eb042faa388eab187c44556ad"}
Nov 02 14:56:46.155804 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.155776    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal" event={"ID":"2dacd25e67dd69f0add8a7632a4c6959","Type":"ContainerStarted","Data":"2c393909b1183d7d0d16bb4666be3ba86722a2d81e2d96f527bd81c5c0aa6570"}
Nov 02 14:56:46.155985 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.155750    7019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.178504 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.178482    7019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:46.178592 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:46.178526    7019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.179181 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.179165    7019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:46.179255 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:46.179199    7019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.272545 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.272478    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.272545 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.272509    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.272545 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.272527    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4d75b8d1582cef207a6bcc13a46c78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"cf4d75b8d1582cef207a6bcc13a46c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.373067 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.373034    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.373067 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.373067    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.373222 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.373088    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4d75b8d1582cef207a6bcc13a46c78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"cf4d75b8d1582cef207a6bcc13a46c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.373222 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.373138    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.373222 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.373157    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2dacd25e67dd69f0add8a7632a4c6959-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" (UID: \"2dacd25e67dd69f0add8a7632a4c6959\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.373222 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.373180    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf4d75b8d1582cef207a6bcc13a46c78-config\") pod \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" (UID: \"cf4d75b8d1582cef207a6bcc13a46c78\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.807911 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.807878    7019 apiserver.go:52] "Watching apiserver"
Nov 02 14:56:46.862363 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.862325    7019 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 14:56:46.863913 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.863874    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0","openshift-monitoring/node-exporter-xhjxz","openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6","openshift-monitoring/thanos-querier-6895fc499c-x7njf","kube-system/global-pull-secret-syncer-l52v6","openshift-cluster-node-tuning-operator/tuned-kzs8g","openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn","openshift-multus/network-metrics-daemon-h5wrn","openshift-dns/dns-default-2nhpf","openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh","openshift-monitoring/metrics-server-64b454d547-2rcbx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj","openshift-image-registry/node-ca-7g9ww","openshift-insights/insights-runtime-extractor-jzm4s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal","openshift-multus/multus-additional-cni-plugins-grmcf","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-xw5bg","kube-system/konnectivity-agent-29z4k","openshift-dns/node-resolver-vpflc","openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9","kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal","openshift-ingress-canary/ingress-canary-wfxpz","openshift-ovn-kubernetes/ovnkube-node-trwcd","openshift-network-operator/iptables-alerter-tbjb6","openshift-network-diagnostics/network-check-target-k67qn"]
Nov 02 14:56:46.881740 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.881712    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 14:56:46.881883 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.881850    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.881942 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.881915    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-25j7m\""
Nov 02 14:56:46.882035 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882020    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 14:56:46.882098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882087    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nnwvk\""
Nov 02 14:56:46.882228 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882215    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-62zsq\""
Nov 02 14:56:46.882364 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882351    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.882454 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882441    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zhczs\""
Nov 02 14:56:46.882523 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882511    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 14:56:46.882616 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882604    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 14:56:46.882685 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882673    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kws2w\""
Nov 02 14:56:46.882802 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882790    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.882852 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882789    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.882909 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882870    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-dddv8\""
Nov 02 14:56:46.882909 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882900    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.883003 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882957    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.883003 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.882975    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.926208 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.926182    7019 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 14:56:46.929218 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.929199    7019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.929364 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.929352    7019 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.953357 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.953332    7019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:46.953471 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:46.953371    7019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-45.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.954163 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.954146    7019 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 14:56:46.954244 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:46.954180    7019 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-45.ec2.internal"
Nov 02 14:56:46.975899 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.975881    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.975988 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.975909    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnz9s\" (UniqueName: \"kubernetes.io/projected/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-kube-api-access-qnz9s\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.975988 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.975928    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-script-lib\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.975988 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.975947    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-textfile\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.975988 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.975972    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.976150 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976012    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-agent-certs\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:56:46.976150 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976037    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6p6\" (UniqueName: \"kubernetes.io/projected/bbfbe419-497a-489b-9109-c52ed3bf0a6e-kube-api-access-ks6p6\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:46.976150 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976054    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10ef5542-79bd-4e48-866d-952fbeb2bad2-iptables-alerter-script\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:56:46.976150 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976071    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-root\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.976306 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976231    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.976306 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976295    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.976398 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976321    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqc8c\" (UniqueName: \"kubernetes.io/projected/11e1276c-fe6b-44e7-8df9-a7925a8cd350-kube-api-access-tqc8c\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.976398 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976333    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.976398 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976349    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-kubernetes\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.976398 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976330    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-agent-certs\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:56:46.976398 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976397    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-host\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.976617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976401    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-textfile\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.976617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976415    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.976617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976417    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10ef5542-79bd-4e48-866d-952fbeb2bad2-iptables-alerter-script\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:56:46.976617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976447    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.976617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976480    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.976617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976511    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-slash\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.976617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976538    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.976841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976629    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.976841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976673    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-konnectivity-ca\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:56:46.976841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976700    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.976841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976717    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-client-certs\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.976841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976736    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e1b988f-c2c6-4844-84ea-1146294c99b8-crio-socket\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.976841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976818    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976845    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992dc344-ae2d-43e9-89df-42011f2e8794-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976863    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976870    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4ab14eae-d5f0-46e3-abda-c817d0e8b66e-konnectivity-ca\") pod \"konnectivity-agent-29z4k\" (UID: \"4ab14eae-d5f0-46e3-abda-c817d0e8b66e\") " pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976906    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-run\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976925    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-system-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976949    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-systemd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.976974    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-netd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977005    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-env-overrides\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.977058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977032    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-grpc-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977111    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-socket-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977133    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977149    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-modprobe-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977165    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-var-lib-kubelet\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977180    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-metrics-server-audit-profiles\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977220    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-ovn\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977240    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-tuned\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977257    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9q75\" (UniqueName: \"kubernetes.io/projected/3558d580-6e93-4b65-bc56-98234fa61562-kube-api-access-c9q75\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977295    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4tjr\" (UniqueName: \"kubernetes.io/projected/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-kube-api-access-t4tjr\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977313    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977340    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977348    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-tuned\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.977389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977355    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-kubelet-config\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977398    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977425    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977454    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-sys-fs\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977536    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977556    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bc4f60-d5c6-4390-a61e-7898d781b4c2-metrics-tls\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977574    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977665    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977702    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpqm\" (UniqueName: \"kubernetes.io/projected/93eca203-b6a7-41d2-b640-953dd6ccaa02-kube-api-access-ggpqm\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977730    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977758    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-cnibin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977784    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bc4f60-d5c6-4390-a61e-7898d781b4c2-tmp-dir\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977813    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-sys\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977840    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-tls\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977867    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-tls\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977893    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wt7\" (UniqueName: \"kubernetes.io/projected/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-kube-api-access-t7wt7\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977938    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-netns\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.977957 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977950    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67bc4f60-d5c6-4390-a61e-7898d781b4c2-tmp-dir\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.977981    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-federate-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978001    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-socket-dir-parent\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978020    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-metrics-client-ca\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978048    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978075    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-node-log\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978103    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978137    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-k8s-cni-cncf-io\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978144    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978165    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978188    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-kubelet\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978204    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-var-lib-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978219    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-registration-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978237    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7dtr\" (UniqueName: \"kubernetes.io/projected/044f0c98-3b8f-4283-9bdf-d235a867e360-kube-api-access-x7dtr\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978333    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-conf-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978365    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-multus-certs\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978396    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv26p\" (UniqueName: \"kubernetes.io/projected/be62e46e-7648-4e29-a2ba-fce4655087ee-kube-api-access-nv26p\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978431    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978459    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ea0a743-d6e8-4211-bd0f-b719dfc91508-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978499    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978526    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-system-cni-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978559    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978563    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ea0a743-d6e8-4211-bd0f-b719dfc91508-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978588    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978625    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.978661 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978665    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978684    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978706    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978723    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-etc-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978740    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovn-node-metrics-cert\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978754    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978763    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978820    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978904    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1e0c29-c679-40f0-9961-7f7ff476e04e-host\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978930    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.978996    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979078    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979109    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e1b988f-c2c6-4844-84ea-1146294c99b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979136    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-netns\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979160    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cnibin\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979189    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8kn\" (UniqueName: \"kubernetes.io/projected/67bc4f60-d5c6-4390-a61e-7898d781b4c2-kube-api-access-9s8kn\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979229    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be62e46e-7648-4e29-a2ba-fce4655087ee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979257    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979284    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979320    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979350    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979380    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsf7\" (UniqueName: \"kubernetes.io/projected/77b573dd-a446-4605-8362-2fd3e4565d3d-kube-api-access-rtsf7\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979407    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89t4\" (UniqueName: \"kubernetes.io/projected/7c1e0c29-c679-40f0-9961-7f7ff476e04e-kube-api-access-n89t4\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979433    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979459    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-log-socket\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979484    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-device-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979512    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbfbe419-497a-489b-9109-c52ed3bf0a6e-cert\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979572    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e1b988f-c2c6-4844-84ea-1146294c99b8-data-volume\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979600    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-bin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979623    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-kubelet\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979661    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e1b988f-c2c6-4844-84ea-1146294c99b8-data-volume\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.979676 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979667    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979712    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9g8\" (UniqueName: \"kubernetes.io/projected/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-api-access-lq9g8\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979731    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93eca203-b6a7-41d2-b640-953dd6ccaa02-tmp-dir\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979748    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-systemd\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979768    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35254468-9d2e-4c69-a162-02ae0854b07f-original-pull-secret\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979770    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-multus\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979802    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bc4f60-d5c6-4390-a61e-7898d781b4c2-config-volume\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979835    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93eca203-b6a7-41d2-b640-953dd6ccaa02-tmp-dir\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979859    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979908    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979927    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-bin\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.979951    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980014    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-conf\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980090    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-multus-daemon-config\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980126    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-etc-kubernetes\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980186    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980217    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93eca203-b6a7-41d2-b640-953dd6ccaa02-hosts-file\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980237    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-sys\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980256    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-lib-modules\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980279    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980303    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980333    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xjr\" (UniqueName: \"kubernetes.io/projected/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-api-access-b7xjr\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980391    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-tmp\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980417    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ppl\" (UniqueName: \"kubernetes.io/projected/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-kube-api-access-c5ppl\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980437    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980448    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77b573dd-a446-4605-8362-2fd3e4565d3d-tmp\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980455    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980485    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980523    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-metrics-client-ca\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980545    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980569    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980596    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-dbus\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980623    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980667    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980689    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-metrics-client-ca\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.980696 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980708    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980735    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1e0c29-c679-40f0-9961-7f7ff476e04e-serviceca\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980760    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980788    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5b4q\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980816    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-audit-log\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980901    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-audit-log\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980903    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980968    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.980999    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981008    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981046    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981085    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-config\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981134    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5t4\" (UniqueName: \"kubernetes.io/projected/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-kube-api-access-dt5t4\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981154    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-wtmp\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981190    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981257    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwm4\" (UniqueName: \"kubernetes.io/projected/1b56d86c-0e63-4e77-8fa7-469ebc91e866-kube-api-access-bhwm4\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981336    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-tmp\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981380    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlz4\" (UniqueName: \"kubernetes.io/projected/10ef5542-79bd-4e48-866d-952fbeb2bad2-kube-api-access-lxlz4\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981402    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981407    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-tmp\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981419    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-client-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981437    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-systemd-units\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981494    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981501    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981521    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10ef5542-79bd-4e48-866d-952fbeb2bad2-host-slash\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981550    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-cni-binary-copy\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981573    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981587    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981588    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981614    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981618    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-hostroot\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981667    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981697    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981699    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981721    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981754    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981751    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfj5v\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981794    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-serving-certs-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981799    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981804    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981799    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981827    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5nf\" (UniqueName: \"kubernetes.io/projected/992dc344-ae2d-43e9-89df-42011f2e8794-kube-api-access-2p5nf\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.981854    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysconfig\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:46.982088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982048    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982120    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982202    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982458    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982460    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982700    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982744    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wnnhb\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982751    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dlg54\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982772    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1hearnjc8vakj\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982858    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982897    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982918    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eqr3qis59nkhr\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982932    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.982986    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f8vff\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983001    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983052    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983082    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983094    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983100    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983100    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983119    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983138    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ssfbf\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983144    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x78pf\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983166    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2fmch\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983192    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983297    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983308    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983304    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983326    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983343    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983347    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983369    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983380    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jmg9z\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983382    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-st89p\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983382    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983460    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983468    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983521    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983560    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pj6nr\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983580    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983582    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983590    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983612    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983628    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983637    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983641    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983681    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4rlsn\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983686    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983640    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983681    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 14:56:46.983717 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983726    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983770    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983819    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983953    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983972    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.983982    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984013    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984028    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984065    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984083    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984102    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984082    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984107    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984135    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984148    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984149    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pvkf\""
Nov 02 14:56:46.985234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.984186    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5g5zn\""
Nov 02 14:56:46.986522 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.986506    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.986599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.986559    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.986730 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.986700    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.986730 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.986714    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987519    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987548    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987576    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987690    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-client-certs\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987696    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987712    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-metrics-server-audit-profiles\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987718    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987705    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987825    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.987854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987851    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/67bc4f60-d5c6-4390-a61e-7898d781b4c2-metrics-tls\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:46.988199 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987701    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-grpc-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.988199 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987524    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.988199 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.987983    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.988199 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.988013    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-metrics-certs\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:46.988199 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.988058    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-env-overrides\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.988199 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.988110    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-secret-metrics-server-tls\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:46.988361 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.988233    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-federate-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989094    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989099    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989125    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989142    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989151    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovn-node-metrics-cert\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989170    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989188    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989193    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.989225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989188    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.989498 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.989277    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990480    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-secret-telemeter-client\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990488    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990499    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/67bc4f60-d5c6-4390-a61e-7898d781b4c2-config-volume\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990522    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990530    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-secret-thanos-querier-tls\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990537    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-multus-daemon-config\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990543    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:46.990599 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990564    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bbfbe419-497a-489b-9109-c52ed3bf0a6e-cert\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:46.990938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990670    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be62e46e-7648-4e29-a2ba-fce4655087ee-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:46.990938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990686    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.990938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990688    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.990938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990702    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cni-binary-copy\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:46.990938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990795    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-client-tls\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.990938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990811    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.991163 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990962    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/992dc344-ae2d-43e9-89df-42011f2e8794-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:46.991163 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.990997    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.991163 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991071    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.991277 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991165    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.991277 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991171    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.991277 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991251    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-config\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.991395 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991378    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c1e0c29-c679-40f0-9961-7f7ff476e04e-serviceca\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:56:46.991786 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991772    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3558d580-6e93-4b65-bc56-98234fa61562-cni-binary-copy\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:46.991902 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991891    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.991988 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991975    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.992031 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.991996    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-serving-certs-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:46.992600 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.992587    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-btkxl\""
Nov 02 14:56:46.992684 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.992673    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7sjw\""
Nov 02 14:56:46.992719 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.992693    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 14:56:46.993838 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.993823    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-62xwk\""
Nov 02 14:56:46.993914 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.993865    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 14:56:46.993960 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.993923    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 14:56:46.993992 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.993975    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nts94\""
Nov 02 14:56:46.993992 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.993983    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hckbz\""
Nov 02 14:56:46.994110 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.994097    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4qhtv\""
Nov 02 14:56:46.994168 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.994129    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aha88bqjs45nv\""
Nov 02 14:56:46.994300 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.994287    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 14:56:46.994346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.994330    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 14:56:46.994395 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.994345    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 14:56:46.994395 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.994383    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 14:56:46.996391 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.996213    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 14:56:46.998192 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.997837    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/992dc344-ae2d-43e9-89df-42011f2e8794-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:46.998192 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.998027    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.998192 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.998041    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1b56d86c-0e63-4e77-8fa7-469ebc91e866-ovnkube-script-lib\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:46.998351 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.998221    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-tls\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.998351 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.998332    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-metrics-client-ca\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.998437 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.998398    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:46.999341 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.999294    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:46.999743 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.999725    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e1b988f-c2c6-4844-84ea-1146294c99b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:46.999838 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.999820    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-accelerators-collector-config\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:46.999888 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:46.999840    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:47.000527 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.000511    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:47.001727 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.001707    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-metrics-client-ca\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:47.001727 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.001722    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-metrics-client-ca\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:47.001871 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.001742    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-client-ca-bundle\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:47.006401 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.006380    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 14:56:47.006497 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.006404    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 14:56:47.006497 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.006414    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.006623    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007107    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnz9s\" (UniqueName: \"kubernetes.io/projected/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-kube-api-access-qnz9s\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007129    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqc8c\" (UniqueName: \"kubernetes.io/projected/11e1276c-fe6b-44e7-8df9-a7925a8cd350-kube-api-access-tqc8c\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007127    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpqm\" (UniqueName: \"kubernetes.io/projected/93eca203-b6a7-41d2-b640-953dd6ccaa02-kube-api-access-ggpqm\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007221    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9q75\" (UniqueName: \"kubernetes.io/projected/3558d580-6e93-4b65-bc56-98234fa61562-kube-api-access-c9q75\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007337    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6p6\" (UniqueName: \"kubernetes.io/projected/bbfbe419-497a-489b-9109-c52ed3bf0a6e-kube-api-access-ks6p6\") pod \"ingress-canary-wfxpz\" (UID: \"bbfbe419-497a-489b-9109-c52ed3bf0a6e\") " pod="openshift-ingress-canary/ingress-canary-wfxpz"
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007364    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wt7\" (UniqueName: \"kubernetes.io/projected/3d3e864f-07ce-47ff-a6f5-e4197c15f72d-kube-api-access-t7wt7\") pod \"metrics-server-64b454d547-2rcbx\" (UID: \"3d3e864f-07ce-47ff-a6f5-e4197c15f72d\") " pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:47.007733 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007369    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ps9\" (UniqueName: \"kubernetes.io/projected/ce469d24-f9cc-4381-be8a-5c9006c21db7-kube-api-access-65ps9\") pod \"network-check-target-k67qn\" (UID: \"ce469d24-f9cc-4381-be8a-5c9006c21db7\") " pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:47.008052 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007792    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsf7\" (UniqueName: \"kubernetes.io/projected/77b573dd-a446-4605-8362-2fd3e4565d3d-kube-api-access-rtsf7\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.008052 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007842    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89t4\" (UniqueName: \"kubernetes.io/projected/7c1e0c29-c679-40f0-9961-7f7ff476e04e-kube-api-access-n89t4\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:56:47.008052 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007851    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8kn\" (UniqueName: \"kubernetes.io/projected/67bc4f60-d5c6-4390-a61e-7898d781b4c2-kube-api-access-9s8kn\") pod \"dns-default-2nhpf\" (UID: \"67bc4f60-d5c6-4390-a61e-7898d781b4c2\") " pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:47.008052 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007886    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv26p\" (UniqueName: \"kubernetes.io/projected/be62e46e-7648-4e29-a2ba-fce4655087ee-kube-api-access-nv26p\") pod \"monitoring-plugin-77f4f76594-bqrd9\" (UID: \"be62e46e-7648-4e29-a2ba-fce4655087ee\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:47.008052 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007901    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4tjr\" (UniqueName: \"kubernetes.io/projected/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-kube-api-access-t4tjr\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:47.008052 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.007909    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9g8\" (UniqueName: \"kubernetes.io/projected/8ea0a743-d6e8-4211-bd0f-b719dfc91508-kube-api-access-lq9g8\") pod \"kube-state-metrics-5dc8d7d89b-lxnwh\" (UID: \"8ea0a743-d6e8-4211-bd0f-b719dfc91508\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lxnwh"
Nov 02 14:56:47.008451 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.008097    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7dtr\" (UniqueName: \"kubernetes.io/projected/044f0c98-3b8f-4283-9bdf-d235a867e360-kube-api-access-x7dtr\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.011084 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.011064    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:47.012346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.012322    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:47.012346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.012333    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11e1276c-fe6b-44e7-8df9-a7925a8cd350-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d9f77bcdf-hh5bn\" (UID: \"11e1276c-fe6b-44e7-8df9-a7925a8cd350\") " pod="openshift-monitoring/telemeter-client-5d9f77bcdf-hh5bn"
Nov 02 14:56:47.013163 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.013144    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:47.014088 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.014068    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xjr\" (UniqueName: \"kubernetes.io/projected/2e1b988f-c2c6-4844-84ea-1146294c99b8-kube-api-access-b7xjr\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:47.014151 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.014130    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5t4\" (UniqueName: \"kubernetes.io/projected/e34c8a6c-4225-4d6f-9886-9f1f908d2f5a-kube-api-access-dt5t4\") pod \"thanos-querier-6895fc499c-x7njf\" (UID: \"e34c8a6c-4225-4d6f-9886-9f1f908d2f5a\") " pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:47.014390 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.014374    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ppl\" (UniqueName: \"kubernetes.io/projected/4f1313c9-5c99-44a9-914f-3bdbcfd9a23f-kube-api-access-c5ppl\") pod \"network-metrics-daemon-h5wrn\" (UID: \"4f1313c9-5c99-44a9-914f-3bdbcfd9a23f\") " pod="openshift-multus/network-metrics-daemon-h5wrn"
Nov 02 14:56:47.015085 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.015070    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlz4\" (UniqueName: \"kubernetes.io/projected/10ef5542-79bd-4e48-866d-952fbeb2bad2-kube-api-access-lxlz4\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:56:47.016127 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.016104    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5nf\" (UniqueName: \"kubernetes.io/projected/992dc344-ae2d-43e9-89df-42011f2e8794-kube-api-access-2p5nf\") pod \"openshift-state-metrics-7bd7489957-7pbh6\" (UID: \"992dc344-ae2d-43e9-89df-42011f2e8794\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-7pbh6"
Nov 02 14:56:47.016339 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.016326    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5b4q\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\") pod \"alertmanager-main-0\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:47.016430 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.016415    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfj5v\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\") pod \"prometheus-k8s-0\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:47.016936 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.016918    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwm4\" (UniqueName: \"kubernetes.io/projected/1b56d86c-0e63-4e77-8fa7-469ebc91e866-kube-api-access-bhwm4\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082205 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082137    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-wtmp\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:47.082205 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082171    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-systemd-units\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082205 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082192    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10ef5542-79bd-4e48-866d-952fbeb2bad2-host-slash\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082215    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-hostroot\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082240    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysconfig\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082249    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-systemd-units\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082259    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10ef5542-79bd-4e48-866d-952fbeb2bad2-host-slash\") pod \"iptables-alerter-tbjb6\" (UID: \"10ef5542-79bd-4e48-866d-952fbeb2bad2\") " pod="openshift-network-operator/iptables-alerter-tbjb6"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082283    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-hostroot\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082262    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082308    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-node-exporter-wtmp\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082313    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082317    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysconfig\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082340    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-root\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082359    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-kubernetes\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082373    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-host\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082381    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-root\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:47.082400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082390    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-slash\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082417    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-slash\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082414    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082440    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082422    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-host\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082444    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-kubernetes\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082459    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e1b988f-c2c6-4844-84ea-1146294c99b8-crio-socket\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082481    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-run\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082521    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-run\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082511    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-system-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082559    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-systemd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082563    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e1b988f-c2c6-4844-84ea-1146294c99b8-crio-socket\") pod \"insights-runtime-extractor-jzm4s\" (UID: \"2e1b988f-c2c6-4844-84ea-1146294c99b8\") " pod="openshift-insights/insights-runtime-extractor-jzm4s"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082575    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-netd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082593    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-socket-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082613    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-systemd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082617    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-modprobe-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082634    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-netd\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082641    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-var-lib-kubelet\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082673    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-system-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082683    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-ovn\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082676    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-socket-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082701    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-var-lib-kubelet\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082706    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082723    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-ovn\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082728    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-kubelet-config\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082755    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082766    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-sys-fs\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082759    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-modprobe-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082767    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-kubelet-config\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082797    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082824    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-sys-fs\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082827    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-cnibin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082853    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-sys\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082868    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-netns\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082871    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-cnibin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082873    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-cni-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082883    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-socket-dir-parent\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082902    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3-sys\") pod \"node-exporter-xhjxz\" (UID: \"a14a5cf5-20d5-47c5-bdf8-bccfe1ffc6f3\") " pod="openshift-monitoring/node-exporter-xhjxz"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082903    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-node-log\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082921    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-node-log\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082926    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-netns\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082931    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-k8s-cni-cncf-io\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082949    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-k8s-cni-cncf-io\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082952    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-socket-dir-parent\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082970    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-kubelet\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.082969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.082995    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-var-lib-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083017    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-registration-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083044    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-conf-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083041    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-kubelet\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083055    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-var-lib-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083070    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-multus-certs\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083077    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-registration-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083088    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-multus-conf-dir\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083098    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-system-cni-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083116    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-multus-certs\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083123    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083129    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-system-cni-dir\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083153    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-etc-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083179    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083200    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1e0c29-c679-40f0-9961-7f7ff476e04e-host\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083223    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083227    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-etc-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083246    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-netns\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083257    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c1e0c29-c679-40f0-9961-7f7ff476e04e-host\") pod \"node-ca-7g9ww\" (UID: \"7c1e0c29-c679-40f0-9961-7f7ff476e04e\") " pod="openshift-image-registry/node-ca-7g9ww"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083264    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-run-openvswitch\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083249    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-etc-selinux\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083271    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cnibin\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083279    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-run-netns\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083291    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-d\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083308    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e7d86c9-eec3-46af-91eb-e1a2851a59d4-cnibin\") pod \"multus-additional-cni-plugins-grmcf\" (UID: \"8e7d86c9-eec3-46af-91eb-e1a2851a59d4\") " pod="openshift-multus/multus-additional-cni-plugins-grmcf"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083319    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-log-socket\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083335    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-device-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083351    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-bin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083367    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-kubelet\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083375    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-log-socket\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083383    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-systemd\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083411    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-multus\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083418    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044f0c98-3b8f-4283-9bdf-d235a867e360-device-dir\") pod \"aws-ebs-csi-driver-node-ccmrj\" (UID: \"044f0c98-3b8f-4283-9bdf-d235a867e360\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ccmrj"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083425    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-kubelet\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083421    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-systemd\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083437    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-bin\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083438    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-bin\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083455    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-host-var-lib-cni-multus\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083469    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-cni-bin\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083476    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-conf\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083495    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-etc-kubernetes\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083510    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93eca203-b6a7-41d2-b640-953dd6ccaa02-hosts-file\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083525    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3558d580-6e93-4b65-bc56-98234fa61562-etc-kubernetes\") pod \"multus-xw5bg\" (UID: \"3558d580-6e93-4b65-bc56-98234fa61562\") " pod="openshift-multus/multus-xw5bg"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083531    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-sys\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083546    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-lib-modules\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083557    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/93eca203-b6a7-41d2-b640-953dd6ccaa02-hosts-file\") pod \"node-resolver-vpflc\" (UID: \"93eca203-b6a7-41d2-b640-953dd6ccaa02\") " pod="openshift-dns/node-resolver-vpflc"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083565    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083567    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-etc-sysctl-conf\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083588    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-dbus\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083597    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-sys\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083599    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b56d86c-0e63-4e77-8fa7-469ebc91e866-host-run-ovn-kubernetes\") pod \"ovnkube-node-trwcd\" (UID: \"1b56d86c-0e63-4e77-8fa7-469ebc91e866\") " pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083663    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77b573dd-a446-4605-8362-2fd3e4565d3d-lib-modules\") pod \"tuned-kzs8g\" (UID: \"77b573dd-a446-4605-8362-2fd3e4565d3d\") " pod="openshift-cluster-node-tuning-operator/tuned-kzs8g"
Nov 02 14:56:47.084397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.083674    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35254468-9d2e-4c69-a162-02ae0854b07f-dbus\") pod \"global-pull-secret-syncer-l52v6\" (UID: \"35254468-9d2e-4c69-a162-02ae0854b07f\") " pod="kube-system/global-pull-secret-syncer-l52v6"
Nov 02 14:56:47.394042 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.394012    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:47.513016 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:47.512972    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:56:48.632905 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:48.632869    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2nhpf"
Nov 02 14:56:49.439465 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:49.439440    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:56:49.450145 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:49.450123    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64b454d547-2rcbx"
Nov 02 14:56:50.705066 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:50.705028    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:50.841368 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:50.841331    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:51.625234 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:51.625200    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6895fc499c-x7njf"
Nov 02 14:56:52.471363 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:52.471324    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:52.618683 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:52.618629    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:52.703296 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:52.703262    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:56:54.686754 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:54.686718    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bqrd9"
Nov 02 14:56:54.977252 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:54.977178    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:54.977411 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:54.977385    7019 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 14:56:56.510389 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:56.510354    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-trwcd"
Nov 02 14:56:57.457434 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.457403    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:57.488515 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.488485    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:57.488928 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.488904    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="alertmanager" containerID="cri-o://d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73" gracePeriod=120
Nov 02 14:56:57.489027 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.488930    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="prom-label-proxy" containerID="cri-o://d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a" gracePeriod=120
Nov 02 14:56:57.489027 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.488962    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy" containerID="cri-o://83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940" gracePeriod=120
Nov 02 14:56:57.489139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.488906    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-metric" containerID="cri-o://bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f" gracePeriod=120
Nov 02 14:56:57.489139 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.488906    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-web" containerID="cri-o://d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e" gracePeriod=120
Nov 02 14:56:57.489232 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.488978    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="config-reloader" containerID="cri-o://ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e" gracePeriod=120
Nov 02 14:56:57.602582 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.602534    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:57.733125 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.733103    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:57.766225 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.766200    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:57.885667 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885623    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.885827 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885675    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.885827 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885699    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.885827 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885726    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.885827 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885746    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.885827 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885783    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886095 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885828    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886095 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885884    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886095 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885940    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886095 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.885982    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886095 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.886012    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886095 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.886042    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886095 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.886079    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\") pod \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\" (UID: \"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231\") "
Nov 02 14:56:57.886457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.886207    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:57.886457 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.886377    7019 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.888493 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.888464    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:57.888810 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.888755    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:57.889247 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.889181    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:56:57.889425 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.889385    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "kube-api-access-h5b4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:57.889425 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.889399    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.889586 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.889322    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.889694 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.889568    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.889813 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.889697    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.890685 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.890638    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:57.890981 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.890964    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.893478 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.893452    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.900533 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.900507    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config" pod "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" (UID: "ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:57.924664 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.924622    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:56:57.978074 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978004    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a" exitCode=0
Nov 02 14:56:57.978074 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978024    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f" exitCode=0
Nov 02 14:56:57.978074 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978030    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940" exitCode=0
Nov 02 14:56:57.978074 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978036    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e" exitCode=0
Nov 02 14:56:57.978074 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978041    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e" exitCode=0
Nov 02 14:56:57.978074 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978046    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73" exitCode=0
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978082    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978084    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"}
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978172    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"}
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978216    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"}
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978234    7019 scope.go:122] "RemoveContainer" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978259    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"}
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978302    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"}
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978347    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"}
Nov 02 14:56:57.978449 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.978387    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231","Type":"ContainerDied","Data":"3facf7fca9c04659b8c62219dcff6e59452a1817f042f3e4f6530ed03886ac26"}
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.986954    7019 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.986976    7019 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-tls-assets\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.986987    7019 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.986996    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-metrics-client-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987005    7019 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987017    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5b4q\" (UniqueName: \"kubernetes.io/projected/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-kube-api-access-h5b4q\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987029    7019 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-main-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987037    7019 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-cluster-tls-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987064    7019 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-web-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987072    7019 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-config-out\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987082    7019 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987096    7019 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231-alertmanager-main-db\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:56:57.987755 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.987440    7019 scope.go:122] "RemoveContainer" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:57.994421 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:57.994401    7019 scope.go:122] "RemoveContainer" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.003428 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.001628    7019 scope.go:122] "RemoveContainer" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.009200 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.009186    7019 scope.go:122] "RemoveContainer" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.016277 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.016261    7019 scope.go:122] "RemoveContainer" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.023266 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.023249    7019 scope.go:122] "RemoveContainer" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.030571 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.030547    7019 scope.go:122] "RemoveContainer" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:58.030926 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:58.030903    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": container with ID starting with d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a not found: ID does not exist" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:58.031042 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.030939    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"} err="failed to get container status \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": rpc error: code = NotFound desc = could not find container \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": container with ID starting with d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a not found: ID does not exist"
Nov 02 14:56:58.031042 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.030989    7019 scope.go:122] "RemoveContainer" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:58.031378 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:58.031343    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": container with ID starting with bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f not found: ID does not exist" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:58.031472 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.031374    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"} err="failed to get container status \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": rpc error: code = NotFound desc = could not find container \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": container with ID starting with bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f not found: ID does not exist"
Nov 02 14:56:58.031472 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.031396    7019 scope.go:122] "RemoveContainer" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.031727 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:58.031704    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": container with ID starting with 83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940 not found: ID does not exist" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.031831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.031732    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"} err="failed to get container status \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": rpc error: code = NotFound desc = could not find container \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": container with ID starting with 83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940 not found: ID does not exist"
Nov 02 14:56:58.031831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.031752    7019 scope.go:122] "RemoveContainer" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.032033 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:58.032008    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": container with ID starting with d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e not found: ID does not exist" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.032098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032043    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"} err="failed to get container status \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": rpc error: code = NotFound desc = could not find container \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": container with ID starting with d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e not found: ID does not exist"
Nov 02 14:56:58.032098 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032063    7019 scope.go:122] "RemoveContainer" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.032345 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:58.032326    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": container with ID starting with ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e not found: ID does not exist" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.032402 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032350    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"} err="failed to get container status \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": rpc error: code = NotFound desc = could not find container \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": container with ID starting with ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e not found: ID does not exist"
Nov 02 14:56:58.032402 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032368    7019 scope.go:122] "RemoveContainer" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.032615 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:58.032598    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": container with ID starting with d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73 not found: ID does not exist" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.032683 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032620    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"} err="failed to get container status \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": rpc error: code = NotFound desc = could not find container \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": container with ID starting with d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73 not found: ID does not exist"
Nov 02 14:56:58.032683 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032634    7019 scope.go:122] "RemoveContainer" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.032818 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032801    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:58.032913 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:56:58.032899    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": container with ID starting with a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675 not found: ID does not exist" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.032953 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032917    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"} err="failed to get container status \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": rpc error: code = NotFound desc = could not find container \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": container with ID starting with a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675 not found: ID does not exist"
Nov 02 14:56:58.032953 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.032933    7019 scope.go:122] "RemoveContainer" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:58.033150 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033135    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"} err="failed to get container status \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": rpc error: code = NotFound desc = could not find container \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": container with ID starting with d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a not found: ID does not exist"
Nov 02 14:56:58.033192 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033150    7019 scope.go:122] "RemoveContainer" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:58.033348 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033331    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"} err="failed to get container status \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": rpc error: code = NotFound desc = could not find container \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": container with ID starting with bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f not found: ID does not exist"
Nov 02 14:56:58.033388 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033348    7019 scope.go:122] "RemoveContainer" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.033521 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033506    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"} err="failed to get container status \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": rpc error: code = NotFound desc = could not find container \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": container with ID starting with 83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940 not found: ID does not exist"
Nov 02 14:56:58.033557 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033521    7019 scope.go:122] "RemoveContainer" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.033757 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033742    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"} err="failed to get container status \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": rpc error: code = NotFound desc = could not find container \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": container with ID starting with d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e not found: ID does not exist"
Nov 02 14:56:58.033801 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033757    7019 scope.go:122] "RemoveContainer" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.033950 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033934    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"} err="failed to get container status \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": rpc error: code = NotFound desc = could not find container \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": container with ID starting with ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e not found: ID does not exist"
Nov 02 14:56:58.033990 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.033950    7019 scope.go:122] "RemoveContainer" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.034136 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034120    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"} err="failed to get container status \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": rpc error: code = NotFound desc = could not find container \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": container with ID starting with d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73 not found: ID does not exist"
Nov 02 14:56:58.034221 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034135    7019 scope.go:122] "RemoveContainer" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.034316 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034300    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"} err="failed to get container status \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": rpc error: code = NotFound desc = could not find container \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": container with ID starting with a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675 not found: ID does not exist"
Nov 02 14:56:58.034353 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034316    7019 scope.go:122] "RemoveContainer" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:58.034498 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034482    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"} err="failed to get container status \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": rpc error: code = NotFound desc = could not find container \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": container with ID starting with d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a not found: ID does not exist"
Nov 02 14:56:58.034538 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034497    7019 scope.go:122] "RemoveContainer" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:58.034706 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034686    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"} err="failed to get container status \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": rpc error: code = NotFound desc = could not find container \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": container with ID starting with bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f not found: ID does not exist"
Nov 02 14:56:58.034752 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034710    7019 scope.go:122] "RemoveContainer" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.034922 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034906    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"} err="failed to get container status \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": rpc error: code = NotFound desc = could not find container \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": container with ID starting with 83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940 not found: ID does not exist"
Nov 02 14:56:58.034962 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.034922    7019 scope.go:122] "RemoveContainer" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.035099 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035084    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"} err="failed to get container status \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": rpc error: code = NotFound desc = could not find container \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": container with ID starting with d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e not found: ID does not exist"
Nov 02 14:56:58.035136 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035098    7019 scope.go:122] "RemoveContainer" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.035282 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035266    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"} err="failed to get container status \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": rpc error: code = NotFound desc = could not find container \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": container with ID starting with ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e not found: ID does not exist"
Nov 02 14:56:58.035325 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035281    7019 scope.go:122] "RemoveContainer" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.035519 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035503    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"} err="failed to get container status \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": rpc error: code = NotFound desc = could not find container \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": container with ID starting with d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73 not found: ID does not exist"
Nov 02 14:56:58.035559 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035520    7019 scope.go:122] "RemoveContainer" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.035808 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035789    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"} err="failed to get container status \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": rpc error: code = NotFound desc = could not find container \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": container with ID starting with a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675 not found: ID does not exist"
Nov 02 14:56:58.035808 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.035808    7019 scope.go:122] "RemoveContainer" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:58.036102 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036020    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"} err="failed to get container status \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": rpc error: code = NotFound desc = could not find container \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": container with ID starting with d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a not found: ID does not exist"
Nov 02 14:56:58.036102 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036042    7019 scope.go:122] "RemoveContainer" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:58.036297 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036281    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"} err="failed to get container status \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": rpc error: code = NotFound desc = could not find container \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": container with ID starting with bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f not found: ID does not exist"
Nov 02 14:56:58.036346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036297    7019 scope.go:122] "RemoveContainer" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.036520 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036505    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"} err="failed to get container status \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": rpc error: code = NotFound desc = could not find container \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": container with ID starting with 83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940 not found: ID does not exist"
Nov 02 14:56:58.036520 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036519    7019 scope.go:122] "RemoveContainer" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.036727 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036711    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"} err="failed to get container status \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": rpc error: code = NotFound desc = could not find container \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": container with ID starting with d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e not found: ID does not exist"
Nov 02 14:56:58.036767 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036727    7019 scope.go:122] "RemoveContainer" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.036913 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036896    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"} err="failed to get container status \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": rpc error: code = NotFound desc = could not find container \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": container with ID starting with ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e not found: ID does not exist"
Nov 02 14:56:58.036913 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.036912    7019 scope.go:122] "RemoveContainer" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.037116 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037101    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"} err="failed to get container status \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": rpc error: code = NotFound desc = could not find container \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": container with ID starting with d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73 not found: ID does not exist"
Nov 02 14:56:58.037116 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037114    7019 scope.go:122] "RemoveContainer" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.037322 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037299    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"} err="failed to get container status \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": rpc error: code = NotFound desc = could not find container \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": container with ID starting with a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675 not found: ID does not exist"
Nov 02 14:56:58.037364 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037323    7019 scope.go:122] "RemoveContainer" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:58.037597 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037574    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"} err="failed to get container status \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": rpc error: code = NotFound desc = could not find container \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": container with ID starting with d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a not found: ID does not exist"
Nov 02 14:56:58.037597 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037595    7019 scope.go:122] "RemoveContainer" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:58.037974 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037949    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"} err="failed to get container status \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": rpc error: code = NotFound desc = could not find container \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": container with ID starting with bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f not found: ID does not exist"
Nov 02 14:56:58.037974 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.037971    7019 scope.go:122] "RemoveContainer" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.038258 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.038240    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"} err="failed to get container status \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": rpc error: code = NotFound desc = could not find container \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": container with ID starting with 83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940 not found: ID does not exist"
Nov 02 14:56:58.038352 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.038259    7019 scope.go:122] "RemoveContainer" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.038542 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.038497    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"} err="failed to get container status \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": rpc error: code = NotFound desc = could not find container \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": container with ID starting with d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e not found: ID does not exist"
Nov 02 14:56:58.038542 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.038517    7019 scope.go:122] "RemoveContainer" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.038788 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.038767    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"} err="failed to get container status \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": rpc error: code = NotFound desc = could not find container \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": container with ID starting with ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e not found: ID does not exist"
Nov 02 14:56:58.038868 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.038789    7019 scope.go:122] "RemoveContainer" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.038868 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.038853    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:58.039058 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039041    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"} err="failed to get container status \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": rpc error: code = NotFound desc = could not find container \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": container with ID starting with d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73 not found: ID does not exist"
Nov 02 14:56:58.039119 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039060    7019 scope.go:122] "RemoveContainer" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.039312 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039293    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"} err="failed to get container status \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": rpc error: code = NotFound desc = could not find container \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": container with ID starting with a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675 not found: ID does not exist"
Nov 02 14:56:58.039386 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039313    7019 scope.go:122] "RemoveContainer" containerID="d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"
Nov 02 14:56:58.039559 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039544    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a"} err="failed to get container status \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": rpc error: code = NotFound desc = could not find container \"d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a\": container with ID starting with d3fdc995dde4ee17cec3f68c3db0fc3767ade4f789014548476d85d1df719f6a not found: ID does not exist"
Nov 02 14:56:58.039618 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039558    7019 scope.go:122] "RemoveContainer" containerID="bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"
Nov 02 14:56:58.039799 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039783    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f"} err="failed to get container status \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": rpc error: code = NotFound desc = could not find container \"bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f\": container with ID starting with bde9a43dc3a3ec04f2dfa10add9270d195a543db8634eb1bac19fa35737c3c0f not found: ID does not exist"
Nov 02 14:56:58.039799 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039798    7019 scope.go:122] "RemoveContainer" containerID="83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"
Nov 02 14:56:58.039998 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039983    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940"} err="failed to get container status \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": rpc error: code = NotFound desc = could not find container \"83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940\": container with ID starting with 83d9d29c1ba0a91ac95ef98352c962383c1277c3d3ce24ceccca7f43eb5e9940 not found: ID does not exist"
Nov 02 14:56:58.039998 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.039997    7019 scope.go:122] "RemoveContainer" containerID="d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"
Nov 02 14:56:58.040202 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.040187    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e"} err="failed to get container status \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": rpc error: code = NotFound desc = could not find container \"d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e\": container with ID starting with d02db5f8fa79e811d0dbe210109c29476e45c61fc1e37013f3ae8adf9a8d039e not found: ID does not exist"
Nov 02 14:56:58.040244 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.040202    7019 scope.go:122] "RemoveContainer" containerID="ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"
Nov 02 14:56:58.040397 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.040380    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e"} err="failed to get container status \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": rpc error: code = NotFound desc = could not find container \"ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e\": container with ID starting with ff1649f53ba02dfd510fbc33ce4fa267f82b83d58af1b5dba7200c8f2a01a55e not found: ID does not exist"
Nov 02 14:56:58.040437 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.040398    7019 scope.go:122] "RemoveContainer" containerID="d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"
Nov 02 14:56:58.040586 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.040572    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73"} err="failed to get container status \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": rpc error: code = NotFound desc = could not find container \"d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73\": container with ID starting with d5c55a55a64c23908c6545133fc7eca3c7063cb851923919ed7e9b619ff10e73 not found: ID does not exist"
Nov 02 14:56:58.040631 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.040586    7019 scope.go:122] "RemoveContainer" containerID="a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"
Nov 02 14:56:58.040883 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.040858    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675"} err="failed to get container status \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": rpc error: code = NotFound desc = could not find container \"a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675\": container with ID starting with a21c70580e28cba36845b469460b309b43c884c74f36a79de63de5ef25516675 not found: ID does not exist"
Nov 02 14:56:58.068486 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068454    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:58.068775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068761    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="init-config-reloader"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068775    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="init-config-reloader"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068786    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="alertmanager"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068791    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="alertmanager"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068798    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="prom-label-proxy"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068803    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="prom-label-proxy"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068811    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="config-reloader"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068815    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="config-reloader"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068824    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068828    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy"
Nov 02 14:56:58.068831 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068835    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-web"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068841    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-web"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068860    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-metric"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068865    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-metric"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068904    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-metric"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068913    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="init-config-reloader"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068918    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="alertmanager"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068924    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="prom-label-proxy"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068930    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="config-reloader"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068936    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy"
Nov 02 14:56:58.069138 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.068941    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" containerName="kube-rbac-proxy-web"
Nov 02 14:56:58.071284 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.071270    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.087281 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.087255    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 14:56:58.087387 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.087366    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 14:56:58.087458 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.087438    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 14:56:58.087514 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.087501    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 14:56:58.087568 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.087516    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 14:56:58.088360 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.087810    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 14:56:58.088360 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.087367    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 14:56:58.088360 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.088260    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 14:56:58.088561 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.088434    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hckbz\""
Nov 02 14:56:58.099158 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.099131    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 14:56:58.100876 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.100854    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:58.189400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189371    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189400 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189404    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189626 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189422    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189626 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189477    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-web-config\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189626 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189503    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81f4a97b-3701-4042-b4a3-886367f54b9e-config-out\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189816 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189639    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189816 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189706    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2hk\" (UniqueName: \"kubernetes.io/projected/81f4a97b-3701-4042-b4a3-886367f54b9e-kube-api-access-8v2hk\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189816 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189752    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4a97b-3701-4042-b4a3-886367f54b9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189949 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189852    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81f4a97b-3701-4042-b4a3-886367f54b9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189949 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189883    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81f4a97b-3701-4042-b4a3-886367f54b9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.189949 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189912    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81f4a97b-3701-4042-b4a3-886367f54b9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.190036 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189963    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.190036 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.189996    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290465 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290382    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290465 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290425    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290465 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290443    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290465 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290472    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-web-config\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290783 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290496    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81f4a97b-3701-4042-b4a3-886367f54b9e-config-out\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290783 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290533    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290783 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290556    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2hk\" (UniqueName: \"kubernetes.io/projected/81f4a97b-3701-4042-b4a3-886367f54b9e-kube-api-access-8v2hk\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290783 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290594    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4a97b-3701-4042-b4a3-886367f54b9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290984 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290863    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81f4a97b-3701-4042-b4a3-886367f54b9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290984 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290908    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81f4a97b-3701-4042-b4a3-886367f54b9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290984 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290866    7019 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 14:56:58.290984 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290938    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81f4a97b-3701-4042-b4a3-886367f54b9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.290984 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.290967    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.291229 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.291005    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.291229 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.291191    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/81f4a97b-3701-4042-b4a3-886367f54b9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.291718 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.291694    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81f4a97b-3701-4042-b4a3-886367f54b9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.291876 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.291852    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4a97b-3701-4042-b4a3-886367f54b9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.293788 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.293765    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.293911 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.293886    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.293989 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.293946    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81f4a97b-3701-4042-b4a3-886367f54b9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.293989 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.293971    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.294125 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.294101    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.294214 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.294099    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.294323 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.294307    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.294603 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.294575    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81f4a97b-3701-4042-b4a3-886367f54b9e-config-out\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.295582 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.295562    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81f4a97b-3701-4042-b4a3-886367f54b9e-web-config\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.314011 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.313989    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2hk\" (UniqueName: \"kubernetes.io/projected/81f4a97b-3701-4042-b4a3-886367f54b9e-kube-api-access-8v2hk\") pod \"alertmanager-main-0\" (UID: \"81f4a97b-3701-4042-b4a3-886367f54b9e\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.390356 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.390327    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 14:56:58.440315 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.440215    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-29z4k"
Nov 02 14:56:58.715501 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.715477    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 14:56:58.716867 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:56:58.716841    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f4a97b_3701_4042_b4a3_886367f54b9e.slice/crio-acf8c63e97274c1528f3902729db0cbf99308cdb7215a6c3f40289245de96400 WatchSource:0}: Error finding container acf8c63e97274c1528f3902729db0cbf99308cdb7215a6c3f40289245de96400: Status 404 returned error can't find the container with id acf8c63e97274c1528f3902729db0cbf99308cdb7215a6c3f40289245de96400
Nov 02 14:56:58.982747 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.982667    7019 generic.go:356] "Generic (PLEG): container finished" podID="81f4a97b-3701-4042-b4a3-886367f54b9e" containerID="703af7874eca00c054cffa23e9c03d8acd6bf13a30bc2b945ffcd9bcf2a6af90" exitCode=0
Nov 02 14:56:58.982918 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.982754    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerDied","Data":"703af7874eca00c054cffa23e9c03d8acd6bf13a30bc2b945ffcd9bcf2a6af90"}
Nov 02 14:56:58.982918 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:58.982851    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerStarted","Data":"acf8c63e97274c1528f3902729db0cbf99308cdb7215a6c3f40289245de96400"}
Nov 02 14:56:59.861832 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:59.860024    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231" path="/var/lib/kubelet/pods/ba2369c0-a3f2-4dd1-b7ff-63cffb9d6231/volumes"
Nov 02 14:56:59.991239 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:59.991203    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerStarted","Data":"3bcc616443265f32f792a0ed208c3269b34707bddf89a67f32246829a70b502e"}
Nov 02 14:56:59.991384 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:59.991275    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerStarted","Data":"a91ea2c59b527e5b07e0de5ac972e351695905bfb0652c6f5d2ef2e5663f2e71"}
Nov 02 14:56:59.991384 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:59.991300    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerStarted","Data":"9fb4de15ee36c66ac2fa9d0aeb306560cc7811f4b8d94806f144855095db24ff"}
Nov 02 14:56:59.991384 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:59.991325    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerStarted","Data":"9d8d2c28011c5f7a6cd3d2a6b947952d7f6ad74672e44dd0846a12398aa53aff"}
Nov 02 14:56:59.991384 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:59.991355    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerStarted","Data":"b157f20494a18a52c06d8f326fa4287171f08d16bbc5fc7481209efbea683eba"}
Nov 02 14:56:59.991384 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:56:59.991378    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"81f4a97b-3701-4042-b4a3-886367f54b9e","Type":"ContainerStarted","Data":"4780b763f13c040015d42e27dd397082257273dfa7fe88646773575621d04164"}
Nov 02 14:57:00.072017 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:00.071951    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.061472079 podStartE2EDuration="2.071935224s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.463145ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:00.059253039 +0000 UTC m=+14.313206664" watchObservedRunningTime="2026-11-02 14:57:00.071935224 +0000 UTC m=+14.325888848"
Nov 02 14:57:01.889189 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:01.889148    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:02.042153 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.042119    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:02.068422 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.068396    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:57:02.068703 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.068674    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="prometheus" containerID="cri-o://474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a" gracePeriod=600
Nov 02 14:57:02.068806 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.068715    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy" containerID="cri-o://5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953" gracePeriod=600
Nov 02 14:57:02.068806 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.068732    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5" gracePeriod=600
Nov 02 14:57:02.068806 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.068738    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="thanos-sidecar" containerID="cri-o://561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7" gracePeriod=600
Nov 02 14:57:02.068806 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.068751    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="config-reloader" containerID="cri-o://eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151" gracePeriod=600
Nov 02 14:57:02.069035 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.068791    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-web" containerID="cri-o://e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb" gracePeriod=600
Nov 02 14:57:02.314483 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.314458    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:02.323700 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323677    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.323780 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323708    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.323780 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323730    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.323780 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323756    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.323932 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323787    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.323932 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323905    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324034 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323955    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324034 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.323985    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324140 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324037    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324140 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324071    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324140 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324101    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324140 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324131    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324134    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:02.324346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324164    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324185    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:02.324346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324199    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324229    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324274    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324346 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324314    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324728 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324363    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324728 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324394    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324728 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324463    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\") pod \"18cfe9b9-6abc-494a-aa25-201041f17bd1\" (UID: \"18cfe9b9-6abc-494a-aa25-201041f17bd1\") "
Nov 02 14:57:02.324878 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324738    7019 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.324878 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.324756    7019 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.325144 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.325007    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:02.325144 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.325083    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.325282    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.326861    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "kube-api-access-nfj5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.326972    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.327199    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.327417    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.327485    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.327539    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.327573    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:02.327775 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.327746    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:02.328297 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.328052    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.328297 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.328162    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.328547 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.328525    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.328628 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.328609    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.329023 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.328997    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.329304 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.329290    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.338881 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.338857    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config" pod "18cfe9b9-6abc-494a-aa25-201041f17bd1" (UID: "18cfe9b9-6abc-494a-aa25-201041f17bd1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:57:02.425604 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425570    7019 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425604 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425606    7019 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-config-out\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425621    7019 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-tls-assets\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425634    7019 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-db\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425674    7019 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-metrics-client-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425691    7019 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425706    7019 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-grpc-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425720    7019 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-metrics-client-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425736    7019 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425750    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfj5v\" (UniqueName: \"kubernetes.io/projected/18cfe9b9-6abc-494a-aa25-201041f17bd1-kube-api-access-nfj5v\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425764    7019 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.425791 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425779    7019 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.426063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425793    7019 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.426063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425807    7019 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/18cfe9b9-6abc-494a-aa25-201041f17bd1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.426063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425822    7019 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.426063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425836    7019 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-web-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.426063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425850    7019 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-secret-kube-rbac-proxy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:02.426063 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:02.425864    7019 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/18cfe9b9-6abc-494a-aa25-201041f17bd1-config\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 14:57:03.009317 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009285    7019 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5" exitCode=0
Nov 02 14:57:03.009317 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009307    7019 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953" exitCode=0
Nov 02 14:57:03.009317 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009316    7019 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb" exitCode=0
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009326    7019 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7" exitCode=0
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009336    7019 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151" exitCode=0
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009344    7019 generic.go:356] "Generic (PLEG): container finished" podID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a" exitCode=0
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009338    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"}
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009432    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"}
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009460    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"}
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009374    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009499    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"}
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009538    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"}
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009556    7019 scope.go:122] "RemoveContainer" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009577    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"}
Nov 02 14:57:03.009836 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.009617    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"18cfe9b9-6abc-494a-aa25-201041f17bd1","Type":"ContainerDied","Data":"6fe1f2d0014a121a570295009b07e8f8f6d1f43660301fede3b9eb325aea4efc"}
Nov 02 14:57:03.018134 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.018116    7019 scope.go:122] "RemoveContainer" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.025589 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.025558    7019 scope.go:122] "RemoveContainer" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.036169 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.036141    7019 scope.go:122] "RemoveContainer" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.042966 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.042948    7019 scope.go:122] "RemoveContainer" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.049390 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.049371    7019 scope.go:122] "RemoveContainer" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.056156 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.056138    7019 scope.go:122] "RemoveContainer" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.064297 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.064143    7019 scope.go:122] "RemoveContainer" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.064835 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:03.064814    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": container with ID starting with 0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5 not found: ID does not exist" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.064918 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.064847    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"} err="failed to get container status \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": rpc error: code = NotFound desc = could not find container \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": container with ID starting with 0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5 not found: ID does not exist"
Nov 02 14:57:03.064918 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.064871    7019 scope.go:122] "RemoveContainer" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.065141 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:03.065126    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": container with ID starting with 5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953 not found: ID does not exist" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.065203 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.065148    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"} err="failed to get container status \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": rpc error: code = NotFound desc = could not find container \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": container with ID starting with 5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953 not found: ID does not exist"
Nov 02 14:57:03.065203 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.065169    7019 scope.go:122] "RemoveContainer" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.065405 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:03.065381    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": container with ID starting with e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb not found: ID does not exist" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.065443 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.065415    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"} err="failed to get container status \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": rpc error: code = NotFound desc = could not find container \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": container with ID starting with e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb not found: ID does not exist"
Nov 02 14:57:03.065443 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.065435    7019 scope.go:122] "RemoveContainer" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.065733 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:03.065715    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": container with ID starting with 561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7 not found: ID does not exist" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.065778 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.065737    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"} err="failed to get container status \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": rpc error: code = NotFound desc = could not find container \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": container with ID starting with 561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7 not found: ID does not exist"
Nov 02 14:57:03.065778 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.065751    7019 scope.go:122] "RemoveContainer" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.066054 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:03.066025    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": container with ID starting with eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151 not found: ID does not exist" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.066054 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066058    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"} err="failed to get container status \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": rpc error: code = NotFound desc = could not find container \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": container with ID starting with eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151 not found: ID does not exist"
Nov 02 14:57:03.066269 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066078    7019 scope.go:122] "RemoveContainer" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.066394 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:03.066372    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": container with ID starting with 474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a not found: ID does not exist" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.066441 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066398    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"} err="failed to get container status \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": rpc error: code = NotFound desc = could not find container \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": container with ID starting with 474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a not found: ID does not exist"
Nov 02 14:57:03.066441 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066417    7019 scope.go:122] "RemoveContainer" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.066670 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:03.066636    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": container with ID starting with 86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7 not found: ID does not exist" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.066758 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066675    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"} err="failed to get container status \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": rpc error: code = NotFound desc = could not find container \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": container with ID starting with 86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7 not found: ID does not exist"
Nov 02 14:57:03.066758 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066690    7019 scope.go:122] "RemoveContainer" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.066936 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066921    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"} err="failed to get container status \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": rpc error: code = NotFound desc = could not find container \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": container with ID starting with 0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5 not found: ID does not exist"
Nov 02 14:57:03.066979 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.066936    7019 scope.go:122] "RemoveContainer" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.067158 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067136    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"} err="failed to get container status \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": rpc error: code = NotFound desc = could not find container \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": container with ID starting with 5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953 not found: ID does not exist"
Nov 02 14:57:03.067203 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067162    7019 scope.go:122] "RemoveContainer" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.067252 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067239    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:57:03.067436 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067421    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"} err="failed to get container status \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": rpc error: code = NotFound desc = could not find container \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": container with ID starting with e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb not found: ID does not exist"
Nov 02 14:57:03.067472 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067437    7019 scope.go:122] "RemoveContainer" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.067668 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067636    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"} err="failed to get container status \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": rpc error: code = NotFound desc = could not find container \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": container with ID starting with 561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7 not found: ID does not exist"
Nov 02 14:57:03.067714 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067669    7019 scope.go:122] "RemoveContainer" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.067894 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067878    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"} err="failed to get container status \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": rpc error: code = NotFound desc = could not find container \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": container with ID starting with eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151 not found: ID does not exist"
Nov 02 14:57:03.067930 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.067894    7019 scope.go:122] "RemoveContainer" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.068106 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068090    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"} err="failed to get container status \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": rpc error: code = NotFound desc = could not find container \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": container with ID starting with 474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a not found: ID does not exist"
Nov 02 14:57:03.068146 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068107    7019 scope.go:122] "RemoveContainer" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.068297 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068282    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"} err="failed to get container status \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": rpc error: code = NotFound desc = could not find container \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": container with ID starting with 86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7 not found: ID does not exist"
Nov 02 14:57:03.068348 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068297    7019 scope.go:122] "RemoveContainer" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.068473 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068459    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"} err="failed to get container status \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": rpc error: code = NotFound desc = could not find container \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": container with ID starting with 0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5 not found: ID does not exist"
Nov 02 14:57:03.068517 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068473    7019 scope.go:122] "RemoveContainer" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.068664 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068630    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"} err="failed to get container status \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": rpc error: code = NotFound desc = could not find container \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": container with ID starting with 5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953 not found: ID does not exist"
Nov 02 14:57:03.068710 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068666    7019 scope.go:122] "RemoveContainer" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.068842 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068828    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"} err="failed to get container status \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": rpc error: code = NotFound desc = could not find container \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": container with ID starting with e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb not found: ID does not exist"
Nov 02 14:57:03.068893 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.068842    7019 scope.go:122] "RemoveContainer" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.069056 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069030    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"} err="failed to get container status \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": rpc error: code = NotFound desc = could not find container \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": container with ID starting with 561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7 not found: ID does not exist"
Nov 02 14:57:03.069056 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069049    7019 scope.go:122] "RemoveContainer" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.069284 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069264    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"} err="failed to get container status \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": rpc error: code = NotFound desc = could not find container \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": container with ID starting with eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151 not found: ID does not exist"
Nov 02 14:57:03.069341 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069285    7019 scope.go:122] "RemoveContainer" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.069488 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069470    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"} err="failed to get container status \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": rpc error: code = NotFound desc = could not find container \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": container with ID starting with 474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a not found: ID does not exist"
Nov 02 14:57:03.069523 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069488    7019 scope.go:122] "RemoveContainer" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.069701 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069682    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"} err="failed to get container status \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": rpc error: code = NotFound desc = could not find container \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": container with ID starting with 86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7 not found: ID does not exist"
Nov 02 14:57:03.069701 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069699    7019 scope.go:122] "RemoveContainer" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.069925 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069909    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"} err="failed to get container status \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": rpc error: code = NotFound desc = could not find container \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": container with ID starting with 0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5 not found: ID does not exist"
Nov 02 14:57:03.069925 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.069925    7019 scope.go:122] "RemoveContainer" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.070113 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070095    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"} err="failed to get container status \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": rpc error: code = NotFound desc = could not find container \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": container with ID starting with 5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953 not found: ID does not exist"
Nov 02 14:57:03.070184 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070114    7019 scope.go:122] "RemoveContainer" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.070338 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070320    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"} err="failed to get container status \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": rpc error: code = NotFound desc = could not find container \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": container with ID starting with e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb not found: ID does not exist"
Nov 02 14:57:03.070380 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070338    7019 scope.go:122] "RemoveContainer" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.070544 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070528    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"} err="failed to get container status \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": rpc error: code = NotFound desc = could not find container \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": container with ID starting with 561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7 not found: ID does not exist"
Nov 02 14:57:03.070606 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070545    7019 scope.go:122] "RemoveContainer" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.070860 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070842    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"} err="failed to get container status \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": rpc error: code = NotFound desc = could not find container \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": container with ID starting with eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151 not found: ID does not exist"
Nov 02 14:57:03.070931 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.070860    7019 scope.go:122] "RemoveContainer" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.071121 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071097    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"} err="failed to get container status \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": rpc error: code = NotFound desc = could not find container \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": container with ID starting with 474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a not found: ID does not exist"
Nov 02 14:57:03.071164 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071123    7019 scope.go:122] "RemoveContainer" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.071373 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071355    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"} err="failed to get container status \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": rpc error: code = NotFound desc = could not find container \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": container with ID starting with 86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7 not found: ID does not exist"
Nov 02 14:57:03.071422 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071374    7019 scope.go:122] "RemoveContainer" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.071601 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071585    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"} err="failed to get container status \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": rpc error: code = NotFound desc = could not find container \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": container with ID starting with 0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5 not found: ID does not exist"
Nov 02 14:57:03.071601 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071600    7019 scope.go:122] "RemoveContainer" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.071824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071805    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"} err="failed to get container status \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": rpc error: code = NotFound desc = could not find container \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": container with ID starting with 5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953 not found: ID does not exist"
Nov 02 14:57:03.071824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.071824    7019 scope.go:122] "RemoveContainer" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.072032 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072017    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"} err="failed to get container status \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": rpc error: code = NotFound desc = could not find container \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": container with ID starting with e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb not found: ID does not exist"
Nov 02 14:57:03.072071 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072032    7019 scope.go:122] "RemoveContainer" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.072261 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072243    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"} err="failed to get container status \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": rpc error: code = NotFound desc = could not find container \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": container with ID starting with 561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7 not found: ID does not exist"
Nov 02 14:57:03.072299 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072263    7019 scope.go:122] "RemoveContainer" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.072465 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072448    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"} err="failed to get container status \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": rpc error: code = NotFound desc = could not find container \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": container with ID starting with eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151 not found: ID does not exist"
Nov 02 14:57:03.072465 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072464    7019 scope.go:122] "RemoveContainer" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.072686 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072669    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"} err="failed to get container status \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": rpc error: code = NotFound desc = could not find container \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": container with ID starting with 474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a not found: ID does not exist"
Nov 02 14:57:03.072686 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072685    7019 scope.go:122] "RemoveContainer" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.072899 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072880    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"} err="failed to get container status \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": rpc error: code = NotFound desc = could not find container \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": container with ID starting with 86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7 not found: ID does not exist"
Nov 02 14:57:03.072944 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.072899    7019 scope.go:122] "RemoveContainer" containerID="0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"
Nov 02 14:57:03.073119 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073101    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5"} err="failed to get container status \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": rpc error: code = NotFound desc = could not find container \"0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5\": container with ID starting with 0ecc9f30f03eea07710a7ad3c3c84d535fb2810fe211251580362ee669348ae5 not found: ID does not exist"
Nov 02 14:57:03.073293 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073130    7019 scope.go:122] "RemoveContainer" containerID="5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"
Nov 02 14:57:03.073371 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073354    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953"} err="failed to get container status \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": rpc error: code = NotFound desc = could not find container \"5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953\": container with ID starting with 5b8fa4e803d5eee500cb8e0ad439b569a299e74bad60b7cce4143cd08fd93953 not found: ID does not exist"
Nov 02 14:57:03.073416 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073372    7019 scope.go:122] "RemoveContainer" containerID="e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"
Nov 02 14:57:03.073590 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073572    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb"} err="failed to get container status \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": rpc error: code = NotFound desc = could not find container \"e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb\": container with ID starting with e425afd4b6814424d60cc511e8e56b3b42fba5c7fd7c23931a3c67a85355e9eb not found: ID does not exist"
Nov 02 14:57:03.073630 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073590    7019 scope.go:122] "RemoveContainer" containerID="561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"
Nov 02 14:57:03.073804 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073787    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7"} err="failed to get container status \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": rpc error: code = NotFound desc = could not find container \"561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7\": container with ID starting with 561696bb992ffb8070a195034b1b49b1be7b92c5c2f235cc9775978c679feeb7 not found: ID does not exist"
Nov 02 14:57:03.073883 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.073806    7019 scope.go:122] "RemoveContainer" containerID="eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"
Nov 02 14:57:03.074025 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.074005    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151"} err="failed to get container status \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": rpc error: code = NotFound desc = could not find container \"eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151\": container with ID starting with eee0825ef97fb27930ad583de8205ed13cad3fdf5d051879a8993e137a98e151 not found: ID does not exist"
Nov 02 14:57:03.074069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.074025    7019 scope.go:122] "RemoveContainer" containerID="474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"
Nov 02 14:57:03.074247 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.074229    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a"} err="failed to get container status \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": rpc error: code = NotFound desc = could not find container \"474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a\": container with ID starting with 474311d8fb039778076ab16cf72b09177a956d1f8cc8acff3b94ab85f328689a not found: ID does not exist"
Nov 02 14:57:03.074317 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.074248    7019 scope.go:122] "RemoveContainer" containerID="86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"
Nov 02 14:57:03.074497 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.074477    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7"} err="failed to get container status \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": rpc error: code = NotFound desc = could not find container \"86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7\": container with ID starting with 86828dc4a97004686945e282c742692b1896ba34a4e1ef34ae33bd601d716bf7 not found: ID does not exist"
Nov 02 14:57:03.079605 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.079586    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:57:03.106995 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.106969    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:57:03.107285 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107272    7019 cpu_manager.go:414] "containerMap: removing container" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy"
Nov 02 14:57:03.107343 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107285    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy"
Nov 02 14:57:03.107343 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107300    7019 cpu_manager.go:414] "containerMap: removing container" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-web"
Nov 02 14:57:03.107343 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107305    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-web"
Nov 02 14:57:03.107343 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107324    7019 cpu_manager.go:414] "containerMap: removing container" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="thanos-sidecar"
Nov 02 14:57:03.107343 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107329    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="thanos-sidecar"
Nov 02 14:57:03.107343 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107341    7019 cpu_manager.go:414] "containerMap: removing container" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="init-config-reloader"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107347    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="init-config-reloader"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107353    7019 cpu_manager.go:414] "containerMap: removing container" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-thanos"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107358    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-thanos"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107366    7019 cpu_manager.go:414] "containerMap: removing container" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="config-reloader"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107370    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="config-reloader"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107381    7019 cpu_manager.go:414] "containerMap: removing container" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="prometheus"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107385    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="prometheus"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107427    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-web"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107437    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="thanos-sidecar"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107443    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="init-config-reloader"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107448    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy-thanos"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107453    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="config-reloader"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107459    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="prometheus"
Nov 02 14:57:03.107531 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.107465    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" containerName="kube-rbac-proxy"
Nov 02 14:57:03.111429 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.111415    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.130757 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.130714    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 14:57:03.130757 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.130755    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 14:57:03.130886 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.130769    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 14:57:03.130944 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.130932    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 14:57:03.131372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131350    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 14:57:03.131444 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131374    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eqr3qis59nkhr\""
Nov 02 14:57:03.131444 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131375    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 14:57:03.131444 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131391    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 14:57:03.131444 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131400    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 14:57:03.131444 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131411    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 14:57:03.131623 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131597    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 14:57:03.131740 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131728    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f8vff\""
Nov 02 14:57:03.131793 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131781    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 14:57:03.131915 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131903    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 14:57:03.132002 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.131986    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132033 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132013    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132062 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132036    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132062 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132053    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132122 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132101    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132214 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132199    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132248 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132238    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-config\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132280 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132255    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132280 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132270    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132333 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132287    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-web-config\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132367 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132341    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvtk\" (UniqueName: \"kubernetes.io/projected/33dfde8f-423a-4376-8109-562305b7108b-kube-api-access-tjvtk\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132409 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132395    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33dfde8f-423a-4376-8109-562305b7108b-config-out\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132438 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132420    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132468 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132438    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132468 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132453    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132543 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132530    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132578 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132566    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132608 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132602    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132666 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132619    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33dfde8f-423a-4376-8109-562305b7108b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.132710 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.132679    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.144479 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.144403    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 14:57:03.144560 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.144537    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 14:57:03.145233 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.145215    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:57:03.233606 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233573    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.233606 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233605    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.233841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233627    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.233841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233672    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.233841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233724    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.233841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233771    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.233841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233811    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-config\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.233841 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233833    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233860    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233882    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-web-config\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233906    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvtk\" (UniqueName: \"kubernetes.io/projected/33dfde8f-423a-4376-8109-562305b7108b-kube-api-access-tjvtk\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233934    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33dfde8f-423a-4376-8109-562305b7108b-config-out\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233961    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.233988    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.234012    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.234061    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.234082    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.234106    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.234135    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33dfde8f-423a-4376-8109-562305b7108b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234675 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.234179    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.234675 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.234497    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.236824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.235468    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.236824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.235776    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.236824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.236011    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.236824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.236460    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.236824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.236667    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.238144 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.237565    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.238144 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.238012    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.238144 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.238094    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.238357 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.238191    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-web-config\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.238357 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.238211    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/33dfde8f-423a-4376-8109-562305b7108b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.238721 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.238637    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-config\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.238898 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.238878    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.239115 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.239079    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.239489 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.239470    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.239664 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.239631    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/33dfde8f-423a-4376-8109-562305b7108b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.239718 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.239683    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.239718 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.239658    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/33dfde8f-423a-4376-8109-562305b7108b-config-out\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.240301 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.240282    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/33dfde8f-423a-4376-8109-562305b7108b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.269828 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.269729    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvtk\" (UniqueName: \"kubernetes.io/projected/33dfde8f-423a-4376-8109-562305b7108b-kube-api-access-tjvtk\") pod \"prometheus-k8s-0\" (UID: \"33dfde8f-423a-4376-8109-562305b7108b\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.429243 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.429195    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:03.757664 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.757620    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 14:57:03.758616 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:57:03.758588    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33dfde8f_423a_4376_8109_562305b7108b.slice/crio-567fa17c331797d1ebab55f7d5e078f45581b25e3d46c6f6bec95f083f9050c5 WatchSource:0}: Error finding container 567fa17c331797d1ebab55f7d5e078f45581b25e3d46c6f6bec95f083f9050c5: Status 404 returned error can't find the container with id 567fa17c331797d1ebab55f7d5e078f45581b25e3d46c6f6bec95f083f9050c5
Nov 02 14:57:03.859334 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:03.859307    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18cfe9b9-6abc-494a-aa25-201041f17bd1" path="/var/lib/kubelet/pods/18cfe9b9-6abc-494a-aa25-201041f17bd1/volumes"
Nov 02 14:57:04.014917 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:04.014831    7019 generic.go:356] "Generic (PLEG): container finished" podID="33dfde8f-423a-4376-8109-562305b7108b" containerID="ab757277408d4c4c458fe111eab8e55ac4a813a63c75b78477cd52033d9c93b1" exitCode=0
Nov 02 14:57:04.015284 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:04.014913    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerDied","Data":"ab757277408d4c4c458fe111eab8e55ac4a813a63c75b78477cd52033d9c93b1"}
Nov 02 14:57:04.015284 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:04.015006    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerStarted","Data":"567fa17c331797d1ebab55f7d5e078f45581b25e3d46c6f6bec95f083f9050c5"}
Nov 02 14:57:05.022431 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.022399    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerStarted","Data":"3a8de14d33bbd1de2634465331a73af0d0a6146a78c2041918010bfa1dde3dab"}
Nov 02 14:57:05.022900 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.022471    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerStarted","Data":"5539b31beb664e42a2b946584a7e320c2b6aa06a1203e16c1504d54b1cbfdcba"}
Nov 02 14:57:05.022900 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.022521    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerStarted","Data":"3d5f9e93f14c277005abb44272eebe6da999ccfb83baa3b1b5c06c5b81152f13"}
Nov 02 14:57:05.022900 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.022568    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerStarted","Data":"dd28873138adb5f615c6f6277d7a9bcd6a4747f544960d4f6b41df247c0f4dfe"}
Nov 02 14:57:05.022900 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.022612    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerStarted","Data":"dded5c9972b3cbb94843b06593e338e51fb8147ba1946a7f461e1aca5fe645f3"}
Nov 02 14:57:05.022900 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.022690    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"33dfde8f-423a-4376-8109-562305b7108b","Type":"ContainerStarted","Data":"30c0269f42e46ec85fc7401bceca5fa501df913816d72285a90b7699b14cbc3b"}
Nov 02 14:57:05.092596 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.092544    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.076921754 podStartE2EDuration="2.092531154s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.6094ms" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:05.092024269 +0000 UTC m=+19.345977904" watchObservedRunningTime="2026-11-02 14:57:05.092531154 +0000 UTC m=+19.346484791"
Nov 02 14:57:05.660529 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.660489    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:05.798895 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:05.798860    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:08.429873 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:08.429839    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:14.311859 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:14.311824    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:14.327511 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:14.327487    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:15.077108 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:15.077080    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 14:57:15.415724 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:15.415686    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:17.939847 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:17.939811    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k67qn"
Nov 02 14:57:22.703131 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:22.703088    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:57:37.426035 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:57:37.425997    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:57:52.704101 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:57:52.704067    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:58:22.705086 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:58:22.705044    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:58:23.109078 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:58:23.108990    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:37.552314 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:58:37.552232    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:47.515226 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:58:47.515185    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:58:52.704269 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:58:52.704225    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:59:07.430272 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:07.430232    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:07.578662 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:07.578611    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:22.704456 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:59:22.704419    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:59:37.670366 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:37.670327    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:43.108732 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:43.108675    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:43.286763 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:43.286725    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:43.458824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:43.458786    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:44.929969 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:44.929935    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:45.125464 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.125419    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:45.894396 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.894366    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"]
Nov 02 14:59:45.896902 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.896883    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:45.915468 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.915452    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-qvj8m\""
Nov 02 14:59:45.929585 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.929539    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 14:59:45.929585 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.929540    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 14:59:45.930496 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.930480    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"]
Nov 02 14:59:45.997940 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:45.997914    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"]
Nov 02 14:59:46.000232 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.000217    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.011204 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.011181    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.011333 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.011316    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.011375 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.011344    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66h5\" (UniqueName: \"kubernetes.io/projected/f698be3a-9ff0-4a16-90d9-cd329aa031de-kube-api-access-d66h5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.022082 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.022062    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"]
Nov 02 14:59:46.024167 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.024150    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8bq5t\""
Nov 02 14:59:46.095167 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.095140    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"]
Nov 02 14:59:46.097588 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.097572    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.111887 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.111867    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzg9\" (UniqueName: \"kubernetes.io/projected/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-kube-api-access-tbzg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.112020 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.111897    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.112020 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.111921    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.112109 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.112088    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.112145 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.112129    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66h5\" (UniqueName: \"kubernetes.io/projected/f698be3a-9ff0-4a16-90d9-cd329aa031de-kube-api-access-d66h5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.112181 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.112165    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.112320 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.112300    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.112358 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.112344    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.115150 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.115134    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-7qqk4\""
Nov 02 14:59:46.128013 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.127992    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"]
Nov 02 14:59:46.138656 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.138624    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66h5\" (UniqueName: \"kubernetes.io/projected/f698be3a-9ff0-4a16-90d9-cd329aa031de-kube-api-access-d66h5\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.210132 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.210062    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 14:59:46.213355 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213333    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.213417 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213382    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.213484 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213467    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2g4p\" (UniqueName: \"kubernetes.io/projected/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-kube-api-access-b2g4p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.213520 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213504    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.213553 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213540    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzg9\" (UniqueName: \"kubernetes.io/projected/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-kube-api-access-tbzg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.213617 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213598    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.213803 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213781    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.213891 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.213872    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.239993 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.239967    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzg9\" (UniqueName: \"kubernetes.io/projected/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-kube-api-access-tbzg9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.314266 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.314236    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 14:59:46.314448 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.314428    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.314506 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.314463    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2g4p\" (UniqueName: \"kubernetes.io/projected/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-kube-api-access-b2g4p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.314506 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.314487    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.314854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.314833    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.314854 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.314851    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.345184 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.345150    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2g4p\" (UniqueName: \"kubernetes.io/projected/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-kube-api-access-b2g4p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.410156 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.410125    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 14:59:46.430664 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.430602    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:46.511001 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.510977    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"]
Nov 02 14:59:46.512636 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:59:46.512610    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf698be3a_9ff0_4a16_90d9_cd329aa031de.slice/crio-d1ba524edc4b19145df0978846c01118ab9325c0e629feb3e2860500a5cb8706 WatchSource:0}: Error finding container d1ba524edc4b19145df0978846c01118ab9325c0e629feb3e2860500a5cb8706: Status 404 returned error can't find the container with id d1ba524edc4b19145df0978846c01118ab9325c0e629feb3e2860500a5cb8706
Nov 02 14:59:46.514672 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.514631    7019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:59:46.608404 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.608378    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"]
Nov 02 14:59:46.609301 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:59:46.609278    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb5be23_ccb0_4115_beeb_3a0e8c3a551b.slice/crio-1addb57d996af120d92c00f21ff3816e56fd8da93532ab4451f978d6c4739c1f WatchSource:0}: Error finding container 1addb57d996af120d92c00f21ff3816e56fd8da93532ab4451f978d6c4739c1f: Status 404 returned error can't find the container with id 1addb57d996af120d92c00f21ff3816e56fd8da93532ab4451f978d6c4739c1f
Nov 02 14:59:46.628997 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.628975    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq" event={"ID":"afb5be23-ccb0-4115-beeb-3a0e8c3a551b","Type":"ContainerStarted","Data":"1addb57d996af120d92c00f21ff3816e56fd8da93532ab4451f978d6c4739c1f"}
Nov 02 14:59:46.630010 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.629991    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7" event={"ID":"f698be3a-9ff0-4a16-90d9-cd329aa031de","Type":"ContainerStarted","Data":"d1ba524edc4b19145df0978846c01118ab9325c0e629feb3e2860500a5cb8706"}
Nov 02 14:59:46.704752 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:46.704727    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"]
Nov 02 14:59:46.705477 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:59:46.705451    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0b8bed_30bc_437d_b228_d7285e2a6b4f.slice/crio-f942bf459f67d0b0a1a8865b024662392ff8fea72746b993cce55d4b4f54a7a5 WatchSource:0}: Error finding container f942bf459f67d0b0a1a8865b024662392ff8fea72746b993cce55d4b4f54a7a5: Status 404 returned error can't find the container with id f942bf459f67d0b0a1a8865b024662392ff8fea72746b993cce55d4b4f54a7a5
Nov 02 14:59:47.637097 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:47.637019    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5" event={"ID":"5f0b8bed-30bc-437d-b228-d7285e2a6b4f","Type":"ContainerStarted","Data":"f942bf459f67d0b0a1a8865b024662392ff8fea72746b993cce55d4b4f54a7a5"}
Nov 02 14:59:47.652619 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:47.652497    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:48.338672 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.338619    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"]
Nov 02 14:59:48.341996 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.341971    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.372941 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.372894    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-56nbv\""
Nov 02 14:59:48.374026 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.374005    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"]
Nov 02 14:59:48.435421 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.435182    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.435421 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.435247    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.435421 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.435292    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68k9\" (UniqueName: \"kubernetes.io/projected/1c2fbf07-d5f0-4759-ad4e-2624916891dc-kube-api-access-q68k9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.535914 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.535880    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.536061 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.535939    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68k9\" (UniqueName: \"kubernetes.io/projected/1c2fbf07-d5f0-4759-ad4e-2624916891dc-kube-api-access-q68k9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.536061 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.536045    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.536311 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.536287    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.536501 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.536446    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.563520 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.563483    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68k9\" (UniqueName: \"kubernetes.io/projected/1c2fbf07-d5f0-4759-ad4e-2624916891dc-kube-api-access-q68k9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.659683 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.659584    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 14:59:48.986242 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:48.985312    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"]
Nov 02 14:59:48.988568 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:59:48.988531    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2fbf07_d5f0_4759_ad4e_2624916891dc.slice/crio-d2ab7a4f871bd3cd116e9abb7787a7318474c13dbfdb257d5f793bc259b4a17d WatchSource:0}: Error finding container d2ab7a4f871bd3cd116e9abb7787a7318474c13dbfdb257d5f793bc259b4a17d: Status 404 returned error can't find the container with id d2ab7a4f871bd3cd116e9abb7787a7318474c13dbfdb257d5f793bc259b4a17d
Nov 02 14:59:49.011536 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:49.011487    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:49.167111 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:49.166979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:49.651417 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:49.651377    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9" event={"ID":"1c2fbf07-d5f0-4759-ad4e-2624916891dc","Type":"ContainerStarted","Data":"d2ab7a4f871bd3cd116e9abb7787a7318474c13dbfdb257d5f793bc259b4a17d"}
Nov 02 14:59:50.173222 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.173186    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"]
Nov 02 14:59:50.181884 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.180958    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.201779 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.201748    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pwrm4\""
Nov 02 14:59:50.201924 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.201778    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 14:59:50.201924 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.201829    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 14:59:50.215031 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.215006    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 14:59:50.216546 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.216473    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"]
Nov 02 14:59:50.251319 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251285    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251515 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251353    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251515 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251389    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251515 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251415    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e54497-7979-492f-8d4c-6b3741b7d437-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251515 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251460    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwvbs\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-kube-api-access-bwvbs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251515 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251483    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e54497-7979-492f-8d4c-6b3741b7d437-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251914 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251538    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251914 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251555    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.251914 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.251571    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.270421 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.270367    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:50.321380 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.321342    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"]
Nov 02 14:59:50.324943 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.324921    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.345910 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.345883    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-qdwkv\""
Nov 02 14:59:50.351548 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.351507    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"]
Nov 02 14:59:50.363069 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363047    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363201 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363077    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363201 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363094    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363201 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363181    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363308 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363231    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363308 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363249    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363308 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363264    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e54497-7979-492f-8d4c-6b3741b7d437-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363308 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363293    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwvbs\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-kube-api-access-bwvbs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.363424 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.363310    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e54497-7979-492f-8d4c-6b3741b7d437-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.364263 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.364238    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.364388 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.364311    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.364461 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.364439    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.364515 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.364506    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.364941 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.364916    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e54497-7979-492f-8d4c-6b3741b7d437-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.368244 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.368227    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.370354 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.370334    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e54497-7979-492f-8d4c-6b3741b7d437-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.392616 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.392588    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.392882 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.392862    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwvbs\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-kube-api-access-bwvbs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-flxcw\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.458824 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.458628    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464225    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464294    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464325    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464353    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464375    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464396    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464421    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wslmk\" (UniqueName: \"kubernetes.io/projected/d7147f7a-95de-4a28-939a-5b0eb44a22cc-kube-api-access-wslmk\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464441    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.464610 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.464457    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.497818 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.497546    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:50.569507 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569311    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wslmk\" (UniqueName: \"kubernetes.io/projected/d7147f7a-95de-4a28-939a-5b0eb44a22cc-kube-api-access-wslmk\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569507 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569370    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569507 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569403    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569507 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569464    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569965 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569530    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569965 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569560    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569965 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569600    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569965 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569642    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.569965 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.569698    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.570112 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.570085    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.570719 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.570331    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.570719 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.570353    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.570719 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.570472    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.570964 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.570823    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.572673 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.572590    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.572818 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.572803    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.597165 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.597107    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wslmk\" (UniqueName: \"kubernetes.io/projected/d7147f7a-95de-4a28-939a-5b0eb44a22cc-kube-api-access-wslmk\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.597165 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.597142    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7147f7a-95de-4a28-939a-5b0eb44a22cc-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl\" (UID: \"d7147f7a-95de-4a28-939a-5b0eb44a22cc\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.600392 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.600325    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:50.649833 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.649473    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:50.830665 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:50.828423    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"]
Nov 02 14:59:50.834762 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:59:50.834723    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e54497_7979_492f_8d4c_6b3741b7d437.slice/crio-a3ff9d1a456b0a74b32efbac89b7bdb16558f9c906f7eceeb8fe325e5d8c73d3 WatchSource:0}: Error finding container a3ff9d1a456b0a74b32efbac89b7bdb16558f9c906f7eceeb8fe325e5d8c73d3: Status 404 returned error can't find the container with id a3ff9d1a456b0a74b32efbac89b7bdb16558f9c906f7eceeb8fe325e5d8c73d3
Nov 02 14:59:51.001771 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:51.001624    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"]
Nov 02 14:59:51.006317 ip-10-0-129-45 kubenswrapper[7019]: W1102 14:59:51.006277    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7147f7a_95de_4a28_939a_5b0eb44a22cc.slice/crio-e0efb0b0303b96edbe72f522dfafd23c15aec47966ef2efde4bb470fe2a0a315 WatchSource:0}: Error finding container e0efb0b0303b96edbe72f522dfafd23c15aec47966ef2efde4bb470fe2a0a315: Status 404 returned error can't find the container with id e0efb0b0303b96edbe72f522dfafd23c15aec47966ef2efde4bb470fe2a0a315
Nov 02 14:59:51.668253 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:51.668209    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl" event={"ID":"d7147f7a-95de-4a28-939a-5b0eb44a22cc","Type":"ContainerStarted","Data":"e0efb0b0303b96edbe72f522dfafd23c15aec47966ef2efde4bb470fe2a0a315"}
Nov 02 14:59:51.672038 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:51.672007    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" event={"ID":"67e54497-7979-492f-8d4c-6b3741b7d437","Type":"ContainerStarted","Data":"a3ff9d1a456b0a74b32efbac89b7bdb16558f9c906f7eceeb8fe325e5d8c73d3"}
Nov 02 14:59:52.738545 ip-10-0-129-45 kubenswrapper[7019]: E1102 14:59:52.738492    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 14:59:53.721715 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:53.721633    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 14:59:56.038378 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.038340    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 14:59:56.038758 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.038438    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 14:59:56.038758 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.038491    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 14:59:56.344823 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.344782    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 14:59:56.344938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.344882    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 14:59:56.344938 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.344923    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 14:59:56.706180 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.706095    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" event={"ID":"67e54497-7979-492f-8d4c-6b3741b7d437","Type":"ContainerStarted","Data":"adef7850b2a6e2f390c47d5cd24c9a2893bef01bbd1e6980717a999ac8c6cbcb"}
Nov 02 14:59:56.708337 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.708314    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq" event={"ID":"afb5be23-ccb0-4115-beeb-3a0e8c3a551b","Type":"ContainerStarted","Data":"c981d9368edce83a035aac326a4c0e12165e562469dc17efc483010498ad7609"}
Nov 02 14:59:56.710911 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.710777    7019 generic.go:356] "Generic (PLEG): container finished" podID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerID="f35a669be3b27cd77ae488d48287076cd9601cc14f468fb3bbb11464d69435de" exitCode=0
Nov 02 14:59:56.710911 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.710865    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7" event={"ID":"f698be3a-9ff0-4a16-90d9-cd329aa031de","Type":"ContainerDied","Data":"f35a669be3b27cd77ae488d48287076cd9601cc14f468fb3bbb11464d69435de"}
Nov 02 14:59:56.713700 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.713668    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl" event={"ID":"d7147f7a-95de-4a28-939a-5b0eb44a22cc","Type":"ContainerStarted","Data":"0bd33d14303e8a9269fe025a33d4143407c2f1c39b8eb0e919cd8a3cf556180f"}
Nov 02 14:59:56.758569 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.758509    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podStartSLOduration=1.251912688 podStartE2EDuration="6.758491542s" totalImagesPullingTime="5.506578854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:59:56.757228059 +0000 UTC m=+191.011181683" watchObservedRunningTime="2026-11-02 14:59:56.758491542 +0000 UTC m=+191.012445165"
Nov 02 14:59:56.902920 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:56.902867    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl" podStartSLOduration=1.87458098 podStartE2EDuration="6.902849641s" totalImagesPullingTime="5.028268661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:59:56.902390399 +0000 UTC m=+191.156344025" watchObservedRunningTime="2026-11-02 14:59:56.902849641 +0000 UTC m=+191.156803266"
Nov 02 14:59:57.498535 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.498444    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:57.504372 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.504346    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:57.650730 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.650702    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:57.655767 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.655741    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:57.720004 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.719961    7019 generic.go:356] "Generic (PLEG): container finished" podID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerID="ac3afe6b6ecd3b51d84bf981f360540123e07577a29215f5f03953a475718bbe" exitCode=0
Nov 02 14:59:57.720183 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.720046    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7" event={"ID":"f698be3a-9ff0-4a16-90d9-cd329aa031de","Type":"ContainerDied","Data":"ac3afe6b6ecd3b51d84bf981f360540123e07577a29215f5f03953a475718bbe"}
Nov 02 14:59:57.721741 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.721718    7019 generic.go:356] "Generic (PLEG): container finished" podID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerID="c981d9368edce83a035aac326a4c0e12165e562469dc17efc483010498ad7609" exitCode=0
Nov 02 14:59:57.721834 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.721792    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq" event={"ID":"afb5be23-ccb0-4115-beeb-3a0e8c3a551b","Type":"ContainerDied","Data":"c981d9368edce83a035aac326a4c0e12165e562469dc17efc483010498ad7609"}
Nov 02 14:59:57.723855 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.723828    7019 generic.go:356] "Generic (PLEG): container finished" podID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerID="d89fafee90815cd6002ac81aeccb94981ba6bee808167f912f43098957c4b407" exitCode=0
Nov 02 14:59:57.723963 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.723877    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5" event={"ID":"5f0b8bed-30bc-437d-b228-d7285e2a6b4f","Type":"ContainerDied","Data":"d89fafee90815cd6002ac81aeccb94981ba6bee808167f912f43098957c4b407"}
Nov 02 14:59:57.724179 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.724152    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:57.724739 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.724203    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:57.725148 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.725129    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl"
Nov 02 14:59:57.725243 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:57.725227    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 14:59:58.730241 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:58.730209    7019 generic.go:356] "Generic (PLEG): container finished" podID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerID="67853732a793f35f125f16a73ebf06c935293a1566834e8566ac23a4b6952dc9" exitCode=0
Nov 02 14:59:58.730669 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:58.730307    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9" event={"ID":"1c2fbf07-d5f0-4759-ad4e-2624916891dc","Type":"ContainerDied","Data":"67853732a793f35f125f16a73ebf06c935293a1566834e8566ac23a4b6952dc9"}
Nov 02 14:59:58.732404 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:58.732381    7019 generic.go:356] "Generic (PLEG): container finished" podID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerID="005d8ced81944c0d183f689a8c901c1b5603387bf67a3769b3f9aadd0962d89d" exitCode=0
Nov 02 14:59:58.732498 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:58.732473    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq" event={"ID":"afb5be23-ccb0-4115-beeb-3a0e8c3a551b","Type":"ContainerDied","Data":"005d8ced81944c0d183f689a8c901c1b5603387bf67a3769b3f9aadd0962d89d"}
Nov 02 14:59:58.734848 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:58.734824    7019 generic.go:356] "Generic (PLEG): container finished" podID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerID="f6207a1468aebef49a3baa3ec0d6bd1725d373e9e072653725879e37de522ea1" exitCode=0
Nov 02 14:59:58.734993 ip-10-0-129-45 kubenswrapper[7019]: I1102 14:59:58.734922    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5" event={"ID":"5f0b8bed-30bc-437d-b228-d7285e2a6b4f","Type":"ContainerDied","Data":"f6207a1468aebef49a3baa3ec0d6bd1725d373e9e072653725879e37de522ea1"}
Nov 02 15:00:02.760884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:02.760841    7019 generic.go:356] "Generic (PLEG): container finished" podID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerID="4d52fe8f07170cc922e7c955d32b023b74024f1de430e905e1bbe5906312d27e" exitCode=0
Nov 02 15:00:02.761280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:02.760988    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9" event={"ID":"1c2fbf07-d5f0-4759-ad4e-2624916891dc","Type":"ContainerDied","Data":"4d52fe8f07170cc922e7c955d32b023b74024f1de430e905e1bbe5906312d27e"}
Nov 02 15:00:06.801964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:06.801884    7019 generic.go:356] "Generic (PLEG): container finished" podID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerID="2c8896250a1ccd2f14b88607d3b621e0c9abb2a5f2e26e7cb1315bcce2a89e28" exitCode=0
Nov 02 15:00:06.802395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:06.802039    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7" event={"ID":"f698be3a-9ff0-4a16-90d9-cd329aa031de","Type":"ContainerDied","Data":"2c8896250a1ccd2f14b88607d3b621e0c9abb2a5f2e26e7cb1315bcce2a89e28"}
Nov 02 15:00:07.809221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.809185    7019 generic.go:356] "Generic (PLEG): container finished" podID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerID="9c0e19acbbecc075f183d395370632a804186418694ddbcddfca7dd9c4315a84" exitCode=0
Nov 02 15:00:07.809614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.809290    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq" event={"ID":"afb5be23-ccb0-4115-beeb-3a0e8c3a551b","Type":"ContainerDied","Data":"9c0e19acbbecc075f183d395370632a804186418694ddbcddfca7dd9c4315a84"}
Nov 02 15:00:07.812220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.812189    7019 generic.go:356] "Generic (PLEG): container finished" podID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerID="48d7c43bf4b0d2c12250c26554fe828cd05badba39ae52b2a3f5be02522644dd" exitCode=0
Nov 02 15:00:07.812370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.812265    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5" event={"ID":"5f0b8bed-30bc-437d-b228-d7285e2a6b4f","Type":"ContainerDied","Data":"48d7c43bf4b0d2c12250c26554fe828cd05badba39ae52b2a3f5be02522644dd"}
Nov 02 15:00:07.956858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.956835    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 15:00:07.961121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.961097    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-bundle\") pod \"f698be3a-9ff0-4a16-90d9-cd329aa031de\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") "
Nov 02 15:00:07.961247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.961150    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-util\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-util\") pod \"f698be3a-9ff0-4a16-90d9-cd329aa031de\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") "
Nov 02 15:00:07.961247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.961203    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f698be3a-9ff0-4a16-90d9-cd329aa031de-kube-api-access-d66h5\" (UniqueName: \"kubernetes.io/projected/f698be3a-9ff0-4a16-90d9-cd329aa031de-kube-api-access-d66h5\") pod \"f698be3a-9ff0-4a16-90d9-cd329aa031de\" (UID: \"f698be3a-9ff0-4a16-90d9-cd329aa031de\") "
Nov 02 15:00:07.963763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.963732    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f698be3a-9ff0-4a16-90d9-cd329aa031de-kube-api-access-d66h5" pod "f698be3a-9ff0-4a16-90d9-cd329aa031de" (UID: "f698be3a-9ff0-4a16-90d9-cd329aa031de"). InnerVolumeSpecName "kube-api-access-d66h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:07.963943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.963923    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-bundle" pod "f698be3a-9ff0-4a16-90d9-cd329aa031de" (UID: "f698be3a-9ff0-4a16-90d9-cd329aa031de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:07.967691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:07.967665    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-util" pod "f698be3a-9ff0-4a16-90d9-cd329aa031de" (UID: "f698be3a-9ff0-4a16-90d9-cd329aa031de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:08.062033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.062003    7019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:08.062033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.062029    7019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f698be3a-9ff0-4a16-90d9-cd329aa031de-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:08.062033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.062039    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d66h5\" (UniqueName: \"kubernetes.io/projected/f698be3a-9ff0-4a16-90d9-cd329aa031de-kube-api-access-d66h5\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:08.818658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.818612    7019 generic.go:356] "Generic (PLEG): container finished" podID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerID="cc157e1a2eacb7bf6e9d6bc344783a476f862a597e36ca85507088f9d608594a" exitCode=0
Nov 02 15:00:08.819069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.818701    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9" event={"ID":"1c2fbf07-d5f0-4759-ad4e-2624916891dc","Type":"ContainerDied","Data":"cc157e1a2eacb7bf6e9d6bc344783a476f862a597e36ca85507088f9d608594a"}
Nov 02 15:00:08.820561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.820542    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7"
Nov 02 15:00:08.820664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.820564    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8ksr7" event={"ID":"f698be3a-9ff0-4a16-90d9-cd329aa031de","Type":"ContainerDied","Data":"d1ba524edc4b19145df0978846c01118ab9325c0e629feb3e2860500a5cb8706"}
Nov 02 15:00:08.820664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.820602    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1ba524edc4b19145df0978846c01118ab9325c0e629feb3e2860500a5cb8706"
Nov 02 15:00:08.969543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.969525    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 15:00:08.972684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:08.972665    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 15:00:09.068247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.068222    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-kube-api-access-tbzg9\" (UniqueName: \"kubernetes.io/projected/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-kube-api-access-tbzg9\") pod \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") "
Nov 02 15:00:09.068416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.068261    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-util\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-util\") pod \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") "
Nov 02 15:00:09.068416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.068292    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-bundle\") pod \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") "
Nov 02 15:00:09.068416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.068402    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-bundle\") pod \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\" (UID: \"afb5be23-ccb0-4115-beeb-3a0e8c3a551b\") "
Nov 02 15:00:09.068587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.068431    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-util\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-util\") pod \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") "
Nov 02 15:00:09.068587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.068516    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-kube-api-access-b2g4p\" (UniqueName: \"kubernetes.io/projected/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-kube-api-access-b2g4p\") pod \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\" (UID: \"5f0b8bed-30bc-437d-b228-d7285e2a6b4f\") "
Nov 02 15:00:09.068916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.068869    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-bundle" pod "5f0b8bed-30bc-437d-b228-d7285e2a6b4f" (UID: "5f0b8bed-30bc-437d-b228-d7285e2a6b4f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:09.069059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.069018    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-bundle" pod "afb5be23-ccb0-4115-beeb-3a0e8c3a551b" (UID: "afb5be23-ccb0-4115-beeb-3a0e8c3a551b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:09.071007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.070981    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-kube-api-access-b2g4p" pod "5f0b8bed-30bc-437d-b228-d7285e2a6b4f" (UID: "5f0b8bed-30bc-437d-b228-d7285e2a6b4f"). InnerVolumeSpecName "kube-api-access-b2g4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:09.071150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.071017    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-kube-api-access-tbzg9" pod "afb5be23-ccb0-4115-beeb-3a0e8c3a551b" (UID: "afb5be23-ccb0-4115-beeb-3a0e8c3a551b"). InnerVolumeSpecName "kube-api-access-tbzg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:09.073785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.073760    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-util" pod "afb5be23-ccb0-4115-beeb-3a0e8c3a551b" (UID: "afb5be23-ccb0-4115-beeb-3a0e8c3a551b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:09.074013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.073989    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-util" pod "5f0b8bed-30bc-437d-b228-d7285e2a6b4f" (UID: "5f0b8bed-30bc-437d-b228-d7285e2a6b4f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:09.170155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.170120    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbzg9\" (UniqueName: \"kubernetes.io/projected/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-kube-api-access-tbzg9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:09.170155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.170150    7019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:09.170155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.170161    7019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:09.170367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.170170    7019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/afb5be23-ccb0-4115-beeb-3a0e8c3a551b-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:09.170367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.170178    7019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:09.170367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.170186    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2g4p\" (UniqueName: \"kubernetes.io/projected/5f0b8bed-30bc-437d-b228-d7285e2a6b4f-kube-api-access-b2g4p\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:09.826548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.826520    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq"
Nov 02 15:00:09.826985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.826520    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmk5kq" event={"ID":"afb5be23-ccb0-4115-beeb-3a0e8c3a551b","Type":"ContainerDied","Data":"1addb57d996af120d92c00f21ff3816e56fd8da93532ab4451f978d6c4739c1f"}
Nov 02 15:00:09.826985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.826642    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1addb57d996af120d92c00f21ff3816e56fd8da93532ab4451f978d6c4739c1f"
Nov 02 15:00:09.828578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.828554    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5" event={"ID":"5f0b8bed-30bc-437d-b228-d7285e2a6b4f","Type":"ContainerDied","Data":"f942bf459f67d0b0a1a8865b024662392ff8fea72746b993cce55d4b4f54a7a5"}
Nov 02 15:00:09.828578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.828574    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7pknl5"
Nov 02 15:00:09.828759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.828595    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f942bf459f67d0b0a1a8865b024662392ff8fea72746b993cce55d4b4f54a7a5"
Nov 02 15:00:09.951672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.951635    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 15:00:09.975921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.975895    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-util\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-util\") pod \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") "
Nov 02 15:00:09.976042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.975960    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2fbf07-d5f0-4759-ad4e-2624916891dc-kube-api-access-q68k9\" (UniqueName: \"kubernetes.io/projected/1c2fbf07-d5f0-4759-ad4e-2624916891dc-kube-api-access-q68k9\") pod \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") "
Nov 02 15:00:09.976042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.975991    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-bundle\") pod \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\" (UID: \"1c2fbf07-d5f0-4759-ad4e-2624916891dc\") "
Nov 02 15:00:09.976659 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.976625    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-bundle" pod "1c2fbf07-d5f0-4759-ad4e-2624916891dc" (UID: "1c2fbf07-d5f0-4759-ad4e-2624916891dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:09.977906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.977886    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2fbf07-d5f0-4759-ad4e-2624916891dc-kube-api-access-q68k9" pod "1c2fbf07-d5f0-4759-ad4e-2624916891dc" (UID: "1c2fbf07-d5f0-4759-ad4e-2624916891dc"). InnerVolumeSpecName "kube-api-access-q68k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:09.981109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:09.981086    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-util" pod "1c2fbf07-d5f0-4759-ad4e-2624916891dc" (UID: "1c2fbf07-d5f0-4759-ad4e-2624916891dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:10.077282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:10.077212    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q68k9\" (UniqueName: \"kubernetes.io/projected/1c2fbf07-d5f0-4759-ad4e-2624916891dc-kube-api-access-q68k9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:10.077282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:10.077237    7019 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-bundle\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:10.077282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:10.077246    7019 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c2fbf07-d5f0-4759-ad4e-2624916891dc-util\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:00:10.835282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:10.835250    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9" event={"ID":"1c2fbf07-d5f0-4759-ad4e-2624916891dc","Type":"ContainerDied","Data":"d2ab7a4f871bd3cd116e9abb7787a7318474c13dbfdb257d5f793bc259b4a17d"}
Nov 02 15:00:10.835282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:10.835269    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cvctr9"
Nov 02 15:00:10.835753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:10.835298    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ab7a4f871bd3cd116e9abb7787a7318474c13dbfdb257d5f793bc259b4a17d"
Nov 02 15:00:16.801250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:16.801219    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:18.502987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:18.502951    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:21.253643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:21.253605    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:21.396786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:21.396750    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:22.373704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.373675    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"]
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374041    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerName="util"
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374053    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerName="util"
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374063    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerName="pull"
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374068    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerName="pull"
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374085    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerName="util"
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374090    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerName="util"
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374096    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerName="extract"
Nov 02 15:00:22.374102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374100    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374109    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374114    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374124    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374129    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374138    7019 cpu_manager.go:414] "containerMap: removing container" podUID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerName="pull"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374142    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerName="pull"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374148    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerName="util"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374154    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerName="util"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374162    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerName="pull"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374169    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerName="pull"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374198    7019 cpu_manager.go:414] "containerMap: removing container" podUID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerName="util"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374204    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerName="util"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374216    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerName="pull"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374223    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerName="pull"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374232    7019 cpu_manager.go:414] "containerMap: removing container" podUID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374238    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374320    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0b8bed-30bc-437d-b228-d7285e2a6b4f" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374336    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb5be23-ccb0-4115-beeb-3a0e8c3a551b" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374343    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2fbf07-d5f0-4759-ad4e-2624916891dc" containerName="extract"
Nov 02 15:00:22.374396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.374349    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f698be3a-9ff0-4a16-90d9-cd329aa031de" containerName="extract"
Nov 02 15:00:22.377489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.377468    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"
Nov 02 15:00:22.398748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.398720    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-52jgs\""
Nov 02 15:00:22.409458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.409438    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:22.409570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.409476    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:22.411246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.411223    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"]
Nov 02 15:00:22.483834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.483803    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r875r\" (UniqueName: \"kubernetes.io/projected/3afbbd3a-0443-4878-8556-172bc7d3e8e0-kube-api-access-r875r\") pod \"authorino-operator-7b7bb4bc77-s4kqd\" (UID: \"3afbbd3a-0443-4878-8556-172bc7d3e8e0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"
Nov 02 15:00:22.585148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.585116    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r875r\" (UniqueName: \"kubernetes.io/projected/3afbbd3a-0443-4878-8556-172bc7d3e8e0-kube-api-access-r875r\") pod \"authorino-operator-7b7bb4bc77-s4kqd\" (UID: \"3afbbd3a-0443-4878-8556-172bc7d3e8e0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"
Nov 02 15:00:22.610373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.610341    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r875r\" (UniqueName: \"kubernetes.io/projected/3afbbd3a-0443-4878-8556-172bc7d3e8e0-kube-api-access-r875r\") pod \"authorino-operator-7b7bb4bc77-s4kqd\" (UID: \"3afbbd3a-0443-4878-8556-172bc7d3e8e0\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"
Nov 02 15:00:22.689234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:22.689161    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"
Nov 02 15:00:22.707488 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:00:22.707452    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:00:23.008350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:23.008325    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"]
Nov 02 15:00:23.010164 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:00:23.010139    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afbbd3a_0443_4878_8556_172bc7d3e8e0.slice/crio-aa0087fb3e1f286f7af6ed180de6566cd3b71a27ccea6da8f273a21c29877b81 WatchSource:0}: Error finding container aa0087fb3e1f286f7af6ed180de6566cd3b71a27ccea6da8f273a21c29877b81: Status 404 returned error can't find the container with id aa0087fb3e1f286f7af6ed180de6566cd3b71a27ccea6da8f273a21c29877b81
Nov 02 15:00:23.821290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:23.821252    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:23.886274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:23.886243    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd" event={"ID":"3afbbd3a-0443-4878-8556-172bc7d3e8e0","Type":"ContainerStarted","Data":"aa0087fb3e1f286f7af6ed180de6566cd3b71a27ccea6da8f273a21c29877b81"}
Nov 02 15:00:25.896532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:25.896496    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd" event={"ID":"3afbbd3a-0443-4878-8556-172bc7d3e8e0","Type":"ContainerStarted","Data":"b9216882231e79f0e098e8e71409678721c78cb22398523ebc79217ba09f4fe8"}
Nov 02 15:00:25.896955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:25.896679    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"
Nov 02 15:00:25.941914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:25.941865    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd" podStartSLOduration=1.55662565 podStartE2EDuration="3.941852407s" totalImagesPullingTime="2.385226757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:25.941237445 +0000 UTC m=+220.195191111" watchObservedRunningTime="2026-11-02 15:00:25.941852407 +0000 UTC m=+220.195806031"
Nov 02 15:00:26.203235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:26.203126    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:26.576326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:26.576231    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:26.759964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:26.759925    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:26.993579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:26.993541    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:27.672040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:27.672006    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:28.323315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:28.323284    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:29.122785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:29.122752    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:29.285409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:29.285375    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:31.274256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:31.274220    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:31.419618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:31.419586    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:32.627016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:32.626987    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:32.775234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:32.775198    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:33.291334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.291292    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"]
Nov 02 15:00:33.293876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.293854    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.306938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.306918    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"]
Nov 02 15:00:33.320073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.320048    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rdv4c\""
Nov 02 15:00:33.384476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.384446    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5de1540f-e807-4dbe-880d-9f927b3818e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-x9prs\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.384476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.384475    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qclgc\" (UniqueName: \"kubernetes.io/projected/5de1540f-e807-4dbe-880d-9f927b3818e4-kube-api-access-qclgc\") pod \"kuadrant-operator-controller-manager-cf8469d7b-x9prs\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.485984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.485940    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5de1540f-e807-4dbe-880d-9f927b3818e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-x9prs\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.485984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.485987    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qclgc\" (UniqueName: \"kubernetes.io/projected/5de1540f-e807-4dbe-880d-9f927b3818e4-kube-api-access-qclgc\") pod \"kuadrant-operator-controller-manager-cf8469d7b-x9prs\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.486339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.486320    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5de1540f-e807-4dbe-880d-9f927b3818e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-x9prs\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.513870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.513849    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qclgc\" (UniqueName: \"kubernetes.io/projected/5de1540f-e807-4dbe-880d-9f927b3818e4-kube-api-access-qclgc\") pod \"kuadrant-operator-controller-manager-cf8469d7b-x9prs\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.606314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.606240    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:33.941093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:33.941061    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"]
Nov 02 15:00:33.954581 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:00:33.954555    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de1540f_e807_4dbe_880d_9f927b3818e4.slice/crio-79565e6bc84e244bfef98ce9ab344def93bf0fbd6020e7db08c1092535bfeaaa WatchSource:0}: Error finding container 79565e6bc84e244bfef98ce9ab344def93bf0fbd6020e7db08c1092535bfeaaa: Status 404 returned error can't find the container with id 79565e6bc84e244bfef98ce9ab344def93bf0fbd6020e7db08c1092535bfeaaa
Nov 02 15:00:34.067578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:34.067545    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:34.616340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:34.616309    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:34.936484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:34.936405    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs" event={"ID":"5de1540f-e807-4dbe-880d-9f927b3818e4","Type":"ContainerStarted","Data":"79565e6bc84e244bfef98ce9ab344def93bf0fbd6020e7db08c1092535bfeaaa"}
Nov 02 15:00:36.432903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:36.432871    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:36.853657 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:36.853557    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:36.903287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:36.903258    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-s4kqd"
Nov 02 15:00:37.835756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:37.835667    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:37.926861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:37.926829    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"]
Nov 02 15:00:37.929583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:37.929562    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"
Nov 02 15:00:37.951922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:37.951896    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gv569\""
Nov 02 15:00:37.971287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:37.968807    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"]
Nov 02 15:00:38.029097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.029062    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsx4q\" (UniqueName: \"kubernetes.io/projected/d457cf34-abad-45c0-884f-871d30378f1b-kube-api-access-bsx4q\") pod \"limitador-operator-controller-manager-5778bb4449-8b92j\" (UID: \"d457cf34-abad-45c0-884f-871d30378f1b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"
Nov 02 15:00:38.130666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.130618    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsx4q\" (UniqueName: \"kubernetes.io/projected/d457cf34-abad-45c0-884f-871d30378f1b-kube-api-access-bsx4q\") pod \"limitador-operator-controller-manager-5778bb4449-8b92j\" (UID: \"d457cf34-abad-45c0-884f-871d30378f1b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"
Nov 02 15:00:38.159469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.159432    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsx4q\" (UniqueName: \"kubernetes.io/projected/d457cf34-abad-45c0-884f-871d30378f1b-kube-api-access-bsx4q\") pod \"limitador-operator-controller-manager-5778bb4449-8b92j\" (UID: \"d457cf34-abad-45c0-884f-871d30378f1b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"
Nov 02 15:00:38.244533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.244502    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"
Nov 02 15:00:38.249563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.249530    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:38.477412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.477321    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:38.599423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.599393    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"]
Nov 02 15:00:38.600236 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:00:38.600210    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd457cf34_abad_45c0_884f_871d30378f1b.slice/crio-104b9e5488ec734f19fc65bf7ee8cdc029bbff8fc1d9486da96015d0f0929c70 WatchSource:0}: Error finding container 104b9e5488ec734f19fc65bf7ee8cdc029bbff8fc1d9486da96015d0f0929c70: Status 404 returned error can't find the container with id 104b9e5488ec734f19fc65bf7ee8cdc029bbff8fc1d9486da96015d0f0929c70
Nov 02 15:00:38.674422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.674373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:38.864013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.863982    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:38.958133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.958101    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs" event={"ID":"5de1540f-e807-4dbe-880d-9f927b3818e4","Type":"ContainerStarted","Data":"7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b"}
Nov 02 15:00:38.958310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.958229    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:38.959527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:38.959508    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j" event={"ID":"d457cf34-abad-45c0-884f-871d30378f1b","Type":"ContainerStarted","Data":"104b9e5488ec734f19fc65bf7ee8cdc029bbff8fc1d9486da96015d0f0929c70"}
Nov 02 15:00:39.039301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.039252    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs" podStartSLOduration=1.709766821 podStartE2EDuration="6.03923823s" totalImagesPullingTime="4.329471409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:39.037514845 +0000 UTC m=+233.291468479" watchObservedRunningTime="2026-11-02 15:00:39.03923823 +0000 UTC m=+233.293191853"
Nov 02 15:00:39.123022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.122924    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:39.724056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.724018    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:39.881781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.881723    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"]
Nov 02 15:00:39.885506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.885478    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:39.906211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.906185    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 15:00:39.919024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.918978    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-46gbv\""
Nov 02 15:00:39.919024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.919006    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 15:00:39.919243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.919028    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 15:00:39.919243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.919084    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 15:00:39.919243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.919107    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 15:00:39.919842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.919817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:39.929809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:39.929781    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"]
Nov 02 15:00:40.048876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.048802    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crqs8\" (UniqueName: \"kubernetes.io/projected/7647fee8-528c-4062-896e-58d1b1938bcd-kube-api-access-crqs8\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.048876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.048864    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.049151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.048954    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.049151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.048990    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7647fee8-528c-4062-896e-58d1b1938bcd-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.049151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.049082    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.049301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.049162    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7647fee8-528c-4062-896e-58d1b1938bcd-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.150527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.150498    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.150731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.150537    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.150731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.150565    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7647fee8-528c-4062-896e-58d1b1938bcd-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.150731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.150612    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.150731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.150642    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7647fee8-528c-4062-896e-58d1b1938bcd-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.150731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.150709    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crqs8\" (UniqueName: \"kubernetes.io/projected/7647fee8-528c-4062-896e-58d1b1938bcd-kube-api-access-crqs8\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.151388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.151359    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7647fee8-528c-4062-896e-58d1b1938bcd-manifests\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.166071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.153323    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-metrics-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.166071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.153370    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-webhook-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.166071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.153528    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7647fee8-528c-4062-896e-58d1b1938bcd-tmp\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.166071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.153740    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7647fee8-528c-4062-896e-58d1b1938bcd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.195231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.195198    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crqs8\" (UniqueName: \"kubernetes.io/projected/7647fee8-528c-4062-896e-58d1b1938bcd-kube-api-access-crqs8\") pod \"opendatahub-operator-controller-manager-66c8cfc56f-cv2b8\" (UID: \"7647fee8-528c-4062-896e-58d1b1938bcd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.202901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.202879    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:40.220363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.220336    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:40.398216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.398181    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:40.570304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.570276    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"]
Nov 02 15:00:40.583401 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:00:40.583373    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7647fee8_528c_4062_896e_58d1b1938bcd.slice/crio-1913c5c68060fe3df6144d907b103b51a9a864ec92d03ab6d4308ad5bd1b3408 WatchSource:0}: Error finding container 1913c5c68060fe3df6144d907b103b51a9a864ec92d03ab6d4308ad5bd1b3408: Status 404 returned error can't find the container with id 1913c5c68060fe3df6144d907b103b51a9a864ec92d03ab6d4308ad5bd1b3408
Nov 02 15:00:40.608544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.608514    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:40.778190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.778113    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:40.969625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.969587    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j" event={"ID":"d457cf34-abad-45c0-884f-871d30378f1b","Type":"ContainerStarted","Data":"8dae358301b267e1ce55c099834f8e8414e38a557fb691f35f3cd1c8979783b8"}
Nov 02 15:00:40.970048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.969688    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"
Nov 02 15:00:40.970926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:40.970905    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8" event={"ID":"7647fee8-528c-4062-896e-58d1b1938bcd","Type":"ContainerStarted","Data":"1913c5c68060fe3df6144d907b103b51a9a864ec92d03ab6d4308ad5bd1b3408"}
Nov 02 15:00:41.025587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:41.025535    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j" podStartSLOduration=2.160835741 podStartE2EDuration="4.025513662s" totalImagesPullingTime="1.864677921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:41.02487924 +0000 UTC m=+235.278832863" watchObservedRunningTime="2026-11-02 15:00:41.025513662 +0000 UTC m=+235.279467280"
Nov 02 15:00:41.808722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:41.808685    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:41.986457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:41.986421    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:42.614052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:42.614018    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:42.781130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:42.781091    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:43.987361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:43.987325    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8" event={"ID":"7647fee8-528c-4062-896e-58d1b1938bcd","Type":"ContainerStarted","Data":"74de8aaffcb60ab57a4638dd805c5817f6b6c4a04fc44ff07a2e0350389777f1"}
Nov 02 15:00:44.993724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:44.993693    7019 generic.go:356] "Generic (PLEG): container finished" podID="7647fee8-528c-4062-896e-58d1b1938bcd" containerID="74de8aaffcb60ab57a4638dd805c5817f6b6c4a04fc44ff07a2e0350389777f1" exitCode=0
Nov 02 15:00:44.994090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:44.993738    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8" event={"ID":"7647fee8-528c-4062-896e-58d1b1938bcd","Type":"ContainerDied","Data":"74de8aaffcb60ab57a4638dd805c5817f6b6c4a04fc44ff07a2e0350389777f1"}
Nov 02 15:00:45.999997 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:45.999968    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8" event={"ID":"7647fee8-528c-4062-896e-58d1b1938bcd","Type":"ContainerStarted","Data":"b0606dddee7fbeaf7c2dc201e169780a80309c1fd9693436df7c78f0fb2ed5a7"}
Nov 02 15:00:46.000403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:46.000123    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:46.096809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:46.096758    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8" podStartSLOduration=3.269240497 podStartE2EDuration="7.09674544s" totalImagesPullingTime="3.330418012s" totalInitContainerRuntime="497.086931ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:46.082185569 +0000 UTC m=+240.336139210" watchObservedRunningTime="2026-11-02 15:00:46.09674544 +0000 UTC m=+240.350699079"
Nov 02 15:00:46.821156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:46.821119    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:46.980939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:46.980903    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:47.796437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:47.796397    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:49.966743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:49.966710    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:00:50.630695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.630661    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"]
Nov 02 15:00:50.635690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.635671    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:50.690456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.690422    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"]
Nov 02 15:00:50.751342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.751309    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrpd\" (UniqueName: \"kubernetes.io/projected/5205492e-f380-48ef-9879-f03eaead7bcb-kube-api-access-hfrpd\") pod \"kuadrant-operator-controller-manager-678d87766f-6q2hn\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:50.751492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.751368    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5205492e-f380-48ef-9879-f03eaead7bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6q2hn\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:50.852846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.852814    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrpd\" (UniqueName: \"kubernetes.io/projected/5205492e-f380-48ef-9879-f03eaead7bcb-kube-api-access-hfrpd\") pod \"kuadrant-operator-controller-manager-678d87766f-6q2hn\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:50.853009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.852868    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5205492e-f380-48ef-9879-f03eaead7bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6q2hn\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:50.853207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.853191    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5205492e-f380-48ef-9879-f03eaead7bcb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6q2hn\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:50.878523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.878495    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrpd\" (UniqueName: \"kubernetes.io/projected/5205492e-f380-48ef-9879-f03eaead7bcb-kube-api-access-hfrpd\") pod \"kuadrant-operator-controller-manager-678d87766f-6q2hn\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:50.947544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:50.947467    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:51.270694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:51.270667    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"]
Nov 02 15:00:51.271567 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:00:51.271539    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5205492e_f380_48ef_9879_f03eaead7bcb.slice/crio-aca25aeb235d47c69e61ec1bf7b6f360271c9c6eef445431b3ad59873b29fdc3 WatchSource:0}: Error finding container aca25aeb235d47c69e61ec1bf7b6f360271c9c6eef445431b3ad59873b29fdc3: Status 404 returned error can't find the container with id aca25aeb235d47c69e61ec1bf7b6f360271c9c6eef445431b3ad59873b29fdc3
Nov 02 15:00:51.978689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:51.978639    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-8b92j"
Nov 02 15:00:52.028951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:52.028917    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn" event={"ID":"5205492e-f380-48ef-9879-f03eaead7bcb","Type":"ContainerStarted","Data":"a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22"}
Nov 02 15:00:52.029137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:52.028971    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn" event={"ID":"5205492e-f380-48ef-9879-f03eaead7bcb","Type":"ContainerStarted","Data":"aca25aeb235d47c69e61ec1bf7b6f360271c9c6eef445431b3ad59873b29fdc3"}
Nov 02 15:00:52.029137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:52.029063    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:00:52.081407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:52.081353    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn" podStartSLOduration=2.081338922 podStartE2EDuration="2.081338922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:52.080474976 +0000 UTC m=+246.334428611" watchObservedRunningTime="2026-11-02 15:00:52.081338922 +0000 UTC m=+246.335292544"
Nov 02 15:00:52.490373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:52.490338    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:52.648898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:52.648867    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:52.703475 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:00:52.703444    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:00:52.964711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:52.964666    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:53.178714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:53.178686    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:53.426748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:53.426716    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:57.006354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:57.006324    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66c8cfc56f-cv2b8"
Nov 02 15:00:59.714247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:59.714208    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:00:59.922618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:00:59.922580    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:00.556898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:00.556859    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:00.702732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:00.702694    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:02.843425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:02.843387    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:02.994356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:02.994318    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:03.036134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.036108    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 15:01:03.147280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.147241    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"]
Nov 02 15:01:03.147620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.147591    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs" podUID="5de1540f-e807-4dbe-880d-9f927b3818e4" containerName="manager" containerID="cri-o://7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b" gracePeriod=10
Nov 02 15:01:03.389572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.389550    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:01:03.474376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.474301    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5de1540f-e807-4dbe-880d-9f927b3818e4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5de1540f-e807-4dbe-880d-9f927b3818e4-extensions-socket-volume\") pod \"5de1540f-e807-4dbe-880d-9f927b3818e4\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") "
Nov 02 15:01:03.474376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.474361    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de1540f-e807-4dbe-880d-9f927b3818e4-kube-api-access-qclgc\" (UniqueName: \"kubernetes.io/projected/5de1540f-e807-4dbe-880d-9f927b3818e4-kube-api-access-qclgc\") pod \"5de1540f-e807-4dbe-880d-9f927b3818e4\" (UID: \"5de1540f-e807-4dbe-880d-9f927b3818e4\") "
Nov 02 15:01:03.474687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.474660    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5de1540f-e807-4dbe-880d-9f927b3818e4-extensions-socket-volume" pod "5de1540f-e807-4dbe-880d-9f927b3818e4" (UID: "5de1540f-e807-4dbe-880d-9f927b3818e4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:01:03.476343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.476323    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de1540f-e807-4dbe-880d-9f927b3818e4-kube-api-access-qclgc" pod "5de1540f-e807-4dbe-880d-9f927b3818e4" (UID: "5de1540f-e807-4dbe-880d-9f927b3818e4"). InnerVolumeSpecName "kube-api-access-qclgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:01:03.489271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.489236    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:03.575620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.575588    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qclgc\" (UniqueName: \"kubernetes.io/projected/5de1540f-e807-4dbe-880d-9f927b3818e4-kube-api-access-qclgc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:01:03.575620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.575614    7019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5de1540f-e807-4dbe-880d-9f927b3818e4-extensions-socket-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:01:03.755169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:03.755081    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:04.090698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.090596    7019 generic.go:356] "Generic (PLEG): container finished" podID="5de1540f-e807-4dbe-880d-9f927b3818e4" containerID="7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b" exitCode=0
Nov 02 15:01:04.090698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.090689    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"
Nov 02 15:01:04.091115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.090688    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs" event={"ID":"5de1540f-e807-4dbe-880d-9f927b3818e4","Type":"ContainerDied","Data":"7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b"}
Nov 02 15:01:04.091115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.090777    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs" event={"ID":"5de1540f-e807-4dbe-880d-9f927b3818e4","Type":"ContainerDied","Data":"79565e6bc84e244bfef98ce9ab344def93bf0fbd6020e7db08c1092535bfeaaa"}
Nov 02 15:01:04.091115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.090800    7019 scope.go:122] "RemoveContainer" containerID="7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b"
Nov 02 15:01:04.099198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.099178    7019 scope.go:122] "RemoveContainer" containerID="7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b"
Nov 02 15:01:04.099470 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:01:04.099451    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b\": container with ID starting with 7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b not found: ID does not exist" containerID="7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b"
Nov 02 15:01:04.099524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.099480    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b"} err="failed to get container status \"7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b\": rpc error: code = NotFound desc = could not find container \"7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b\": container with ID starting with 7992ee0e671a4f96e7e49d069996fe9460179690d9ab824484467fd877ae991b not found: ID does not exist"
Nov 02 15:01:04.159371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.159331    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"]
Nov 02 15:01:04.173249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:04.173221    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-x9prs"]
Nov 02 15:01:05.859144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:05.859111    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de1540f-e807-4dbe-880d-9f927b3818e4" path="/var/lib/kubelet/pods/5de1540f-e807-4dbe-880d-9f927b3818e4/volumes"
Nov 02 15:01:06.201060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:06.201024    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:06.372606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:06.372573    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:11.510996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:11.510965    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:11.656470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:11.656430    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:12.008455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:12.008421    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:12.251162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:12.251119    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:12.518333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:12.518291    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:18.386113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:18.386077    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:18.626232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:18.626193    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:18.928664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:18.928570    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:19.286291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.286216    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:19.454278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.454247    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"]
Nov 02 15:01:19.454801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.454673    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5de1540f-e807-4dbe-880d-9f927b3818e4" containerName="manager"
Nov 02 15:01:19.454801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.454687    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de1540f-e807-4dbe-880d-9f927b3818e4" containerName="manager"
Nov 02 15:01:19.454801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.454762    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de1540f-e807-4dbe-880d-9f927b3818e4" containerName="manager"
Nov 02 15:01:19.458128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.458103    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.483782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.483755    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x4gs5\""
Nov 02 15:01:19.505761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.505731    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"]
Nov 02 15:01:19.524863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.524836    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.524890    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.524922    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.524964    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.525009    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.525027    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:19.525204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.525090    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.525188    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824kf\" (UniqueName: \"kubernetes.io/projected/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-kube-api-access-824kf\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.525218    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.525298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.525253    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626584    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626625    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626669    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626709    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626735    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626757    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626786    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824kf\" (UniqueName: \"kubernetes.io/projected/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-kube-api-access-824kf\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.626833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626820    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.627203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.626856    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.627203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.627128    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.627203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.627183    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.627354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.627238    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.627354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.627263    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.627926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.627904    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.629036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.629013    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.629447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.629424    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.652012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.651985    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.652560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.652538    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-824kf\" (UniqueName: \"kubernetes.io/projected/ee1d3ebd-ce9d-467d-8d29-0f09eafc306b-kube-api-access-824kf\") pod \"data-science-gateway-data-science-gateway-class-7c689579c-cjrwd\" (UID: \"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.731539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.731498    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:19.773351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.773320    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:19.924362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:19.924267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:20.094261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.094235    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"]
Nov 02 15:01:20.094724 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:01:20.094700    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1d3ebd_ce9d_467d_8d29_0f09eafc306b.slice/crio-49086d9d07fb4b5fda5b73d11372365cfad758fb3c891701b5196a67b8f9ad28 WatchSource:0}: Error finding container 49086d9d07fb4b5fda5b73d11372365cfad758fb3c891701b5196a67b8f9ad28: Status 404 returned error can't find the container with id 49086d9d07fb4b5fda5b73d11372365cfad758fb3c891701b5196a67b8f9ad28
Nov 02 15:01:20.096695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.096666    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:01:20.096753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.096730    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:01:20.096792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.096760    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:01:20.109349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.109326    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:20.181114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.181035    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd" event={"ID":"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b","Type":"ContainerStarted","Data":"ef7564602e53c65b2ae12dfd2471a57bce4cd140ec07225084bab92c973be382"}
Nov 02 15:01:20.181114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.181091    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd" event={"ID":"ee1d3ebd-ce9d-467d-8d29-0f09eafc306b","Type":"ContainerStarted","Data":"49086d9d07fb4b5fda5b73d11372365cfad758fb3c891701b5196a67b8f9ad28"}
Nov 02 15:01:20.237895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.237835    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd" podStartSLOduration=1.237816284 podStartE2EDuration="1.237816284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:20.237125152 +0000 UTC m=+274.491078778" watchObservedRunningTime="2026-11-02 15:01:20.237816284 +0000 UTC m=+274.491769912"
Nov 02 15:01:20.774115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:20.774083    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:21.363928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.363881    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:21.449323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.449291    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"]
Nov 02 15:01:21.451990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.451966    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.475267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.475234    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"]
Nov 02 15:01:21.486813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.486795    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g2v6q\""
Nov 02 15:01:21.543249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.543220    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 15:01:21.544626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.544600    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjnw\" (UniqueName: \"kubernetes.io/projected/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-kube-api-access-5bjnw\") pod \"limitador-limitador-74dfcfb894-pw6tm\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.544754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.544682    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-config-file\") pod \"limitador-limitador-74dfcfb894-pw6tm\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.556956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.556923    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:21.566211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.566182    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"]
Nov 02 15:01:21.645241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.645167    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjnw\" (UniqueName: \"kubernetes.io/projected/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-kube-api-access-5bjnw\") pod \"limitador-limitador-74dfcfb894-pw6tm\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.645440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.645244    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-config-file\") pod \"limitador-limitador-74dfcfb894-pw6tm\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.645863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.645846    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-config-file\") pod \"limitador-limitador-74dfcfb894-pw6tm\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.674700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.674675    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjnw\" (UniqueName: \"kubernetes.io/projected/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-kube-api-access-5bjnw\") pod \"limitador-limitador-74dfcfb894-pw6tm\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.751122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.751085    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:21.765993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.765969    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:21.779636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:21.779616    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:22.070299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:22.070230    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"]
Nov 02 15:01:22.073386 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:01:22.073361    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5b602e_f7d2_42d9_991a_d3f1b944bedf.slice/crio-3452028d291dcdc2bca1675a5ff9f5a4bc8589df0562fce8b4c291d7aadba135 WatchSource:0}: Error finding container 3452028d291dcdc2bca1675a5ff9f5a4bc8589df0562fce8b4c291d7aadba135: Status 404 returned error can't find the container with id 3452028d291dcdc2bca1675a5ff9f5a4bc8589df0562fce8b4c291d7aadba135
Nov 02 15:01:22.192194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:22.192157    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm" event={"ID":"8d5b602e-f7d2-42d9-991a-d3f1b944bedf","Type":"ContainerStarted","Data":"3452028d291dcdc2bca1675a5ff9f5a4bc8589df0562fce8b4c291d7aadba135"}
Nov 02 15:01:22.192397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:22.192378    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:22.193426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:22.193403    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7c689579c-cjrwd"
Nov 02 15:01:22.434640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:22.434601    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:22.705934 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:01:22.705853    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:01:22.759933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:22.759903    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:23.010050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.009960    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:23.301923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.301900    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:01:23.309512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.309491    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.328900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.328874    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:01:23.397766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.397730    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:01:23.464063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.464033    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74a06875-fd82-45ab-b150-cffc05814608-config-file\") pod \"limitador-limitador-7c9b87d574-tr9hf\" (UID: \"74a06875-fd82-45ab-b150-cffc05814608\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.464209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.464069    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwnl\" (UniqueName: \"kubernetes.io/projected/74a06875-fd82-45ab-b150-cffc05814608-kube-api-access-jzwnl\") pod \"limitador-limitador-7c9b87d574-tr9hf\" (UID: \"74a06875-fd82-45ab-b150-cffc05814608\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.564634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.564560    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74a06875-fd82-45ab-b150-cffc05814608-config-file\") pod \"limitador-limitador-7c9b87d574-tr9hf\" (UID: \"74a06875-fd82-45ab-b150-cffc05814608\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.564634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.564594    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwnl\" (UniqueName: \"kubernetes.io/projected/74a06875-fd82-45ab-b150-cffc05814608-kube-api-access-jzwnl\") pod \"limitador-limitador-7c9b87d574-tr9hf\" (UID: \"74a06875-fd82-45ab-b150-cffc05814608\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.565196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.565178    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74a06875-fd82-45ab-b150-cffc05814608-config-file\") pod \"limitador-limitador-7c9b87d574-tr9hf\" (UID: \"74a06875-fd82-45ab-b150-cffc05814608\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.601602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.601570    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwnl\" (UniqueName: \"kubernetes.io/projected/74a06875-fd82-45ab-b150-cffc05814608-kube-api-access-jzwnl\") pod \"limitador-limitador-7c9b87d574-tr9hf\" (UID: \"74a06875-fd82-45ab-b150-cffc05814608\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.621543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.621520    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:23.929938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:23.929906    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:01:23.930152 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:01:23.930134    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a06875_fd82_45ab_b150_cffc05814608.slice/crio-715ab777b7714ddebd86c178be2e8d420e735494add4058b17866555abe4b56c WatchSource:0}: Error finding container 715ab777b7714ddebd86c178be2e8d420e735494add4058b17866555abe4b56c: Status 404 returned error can't find the container with id 715ab777b7714ddebd86c178be2e8d420e735494add4058b17866555abe4b56c
Nov 02 15:01:24.203827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:24.203722    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm" event={"ID":"8d5b602e-f7d2-42d9-991a-d3f1b944bedf","Type":"ContainerStarted","Data":"d734a12ea770808ab5ed643bb0dd361d01b1fcd9bada3a6e66510c8f8f648911"}
Nov 02 15:01:24.204266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:24.203836    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:24.205976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:24.205950    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf" event={"ID":"74a06875-fd82-45ab-b150-cffc05814608","Type":"ContainerStarted","Data":"0d797270e8d0f3eb9479212bf59259d25eefae21978c6d2a88736cb3960e096a"}
Nov 02 15:01:24.206083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:24.205999    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf" event={"ID":"74a06875-fd82-45ab-b150-cffc05814608","Type":"ContainerStarted","Data":"715ab777b7714ddebd86c178be2e8d420e735494add4058b17866555abe4b56c"}
Nov 02 15:01:24.206217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:24.206199    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:24.249844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:24.249801    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm" podStartSLOduration=2.078559119 podStartE2EDuration="3.249789702s" totalImagesPullingTime="1.171230583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:24.248438541 +0000 UTC m=+278.502392167" watchObservedRunningTime="2026-11-02 15:01:24.249789702 +0000 UTC m=+278.503743327"
Nov 02 15:01:24.295521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:24.295476    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf" podStartSLOduration=1.295463062 podStartE2EDuration="1.295463062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:24.294822518 +0000 UTC m=+278.548776143" watchObservedRunningTime="2026-11-02 15:01:24.295463062 +0000 UTC m=+278.549416685"
Nov 02 15:01:35.211364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:35.211289    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:35.211364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:35.211361    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"
Nov 02 15:01:35.332986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:35.332959    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"]
Nov 02 15:01:35.333159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:35.333137    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm" podUID="8d5b602e-f7d2-42d9-991a-d3f1b944bedf" containerName="limitador" containerID="cri-o://d734a12ea770808ab5ed643bb0dd361d01b1fcd9bada3a6e66510c8f8f648911" gracePeriod=30
Nov 02 15:01:36.262556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.262411    7019 generic.go:356] "Generic (PLEG): container finished" podID="8d5b602e-f7d2-42d9-991a-d3f1b944bedf" containerID="d734a12ea770808ab5ed643bb0dd361d01b1fcd9bada3a6e66510c8f8f648911" exitCode=0
Nov 02 15:01:36.262556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.262479    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm" event={"ID":"8d5b602e-f7d2-42d9-991a-d3f1b944bedf","Type":"ContainerDied","Data":"d734a12ea770808ab5ed643bb0dd361d01b1fcd9bada3a6e66510c8f8f648911"}
Nov 02 15:01:36.262556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.262529    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm" event={"ID":"8d5b602e-f7d2-42d9-991a-d3f1b944bedf","Type":"ContainerDied","Data":"3452028d291dcdc2bca1675a5ff9f5a4bc8589df0562fce8b4c291d7aadba135"}
Nov 02 15:01:36.262556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.262545    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3452028d291dcdc2bca1675a5ff9f5a4bc8589df0562fce8b4c291d7aadba135"
Nov 02 15:01:36.275738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.275714    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:36.392083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.392048    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-config-file\" (UniqueName: \"kubernetes.io/configmap/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-config-file\") pod \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") "
Nov 02 15:01:36.392237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.392162    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-kube-api-access-5bjnw\" (UniqueName: \"kubernetes.io/projected/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-kube-api-access-5bjnw\") pod \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\" (UID: \"8d5b602e-f7d2-42d9-991a-d3f1b944bedf\") "
Nov 02 15:01:36.392422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.392396    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-config-file" pod "8d5b602e-f7d2-42d9-991a-d3f1b944bedf" (UID: "8d5b602e-f7d2-42d9-991a-d3f1b944bedf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:01:36.394120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.394099    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-kube-api-access-5bjnw" pod "8d5b602e-f7d2-42d9-991a-d3f1b944bedf" (UID: "8d5b602e-f7d2-42d9-991a-d3f1b944bedf"). InnerVolumeSpecName "kube-api-access-5bjnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:01:36.493825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.493793    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjnw\" (UniqueName: \"kubernetes.io/projected/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-kube-api-access-5bjnw\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:01:36.493825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:36.493821    7019 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8d5b602e-f7d2-42d9-991a-d3f1b944bedf-config-file\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:01:37.272216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:37.268103    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"
Nov 02 15:01:37.343428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:37.343395    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"]
Nov 02 15:01:37.357468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:37.357437    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pw6tm"]
Nov 02 15:01:37.859811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:37.859781    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5b602e-f7d2-42d9-991a-d3f1b944bedf" path="/var/lib/kubelet/pods/8d5b602e-f7d2-42d9-991a-d3f1b944bedf/volumes"
Nov 02 15:01:37.916240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:37.916207    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:42.822122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:42.822063    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:47.893304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:47.893272    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:51.351556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:51.351516    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:51.494856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:51.494816    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:51.981860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:51.981827    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"]
Nov 02 15:01:51.982319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:51.982302    7019 cpu_manager.go:414] "containerMap: removing container" podUID="8d5b602e-f7d2-42d9-991a-d3f1b944bedf" containerName="limitador"
Nov 02 15:01:51.982368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:51.982321    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5b602e-f7d2-42d9-991a-d3f1b944bedf" containerName="limitador"
Nov 02 15:01:51.982403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:51.982396    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5b602e-f7d2-42d9-991a-d3f1b944bedf" containerName="limitador"
Nov 02 15:01:51.989181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:51.989153    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.004921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.004896    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"]
Nov 02 15:01:52.017742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.017716    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 15:01:52.017904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.017744    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:01:52.017904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.017791    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:01:52.019749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.019731    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-f4rlf\""
Nov 02 15:01:52.021150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.021127    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7q7m\" (UniqueName: \"kubernetes.io/projected/0d405b00-cf02-4e4c-ae08-53b0f5ab5980-kube-api-access-t7q7m\") pod \"postgres-7cdbf6dc44-jc2wz\" (UID: \"0d405b00-cf02-4e4c-ae08-53b0f5ab5980\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.021293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.021159    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0d405b00-cf02-4e4c-ae08-53b0f5ab5980-data\") pod \"postgres-7cdbf6dc44-jc2wz\" (UID: \"0d405b00-cf02-4e4c-ae08-53b0f5ab5980\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.100655 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.100625    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:01:52.121930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.121902    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7q7m\" (UniqueName: \"kubernetes.io/projected/0d405b00-cf02-4e4c-ae08-53b0f5ab5980-kube-api-access-t7q7m\") pod \"postgres-7cdbf6dc44-jc2wz\" (UID: \"0d405b00-cf02-4e4c-ae08-53b0f5ab5980\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.122070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.121940    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0d405b00-cf02-4e4c-ae08-53b0f5ab5980-data\") pod \"postgres-7cdbf6dc44-jc2wz\" (UID: \"0d405b00-cf02-4e4c-ae08-53b0f5ab5980\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.122318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.122301    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0d405b00-cf02-4e4c-ae08-53b0f5ab5980-data\") pod \"postgres-7cdbf6dc44-jc2wz\" (UID: \"0d405b00-cf02-4e4c-ae08-53b0f5ab5980\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.147146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.147120    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7q7m\" (UniqueName: \"kubernetes.io/projected/0d405b00-cf02-4e4c-ae08-53b0f5ab5980-kube-api-access-t7q7m\") pod \"postgres-7cdbf6dc44-jc2wz\" (UID: \"0d405b00-cf02-4e4c-ae08-53b0f5ab5980\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.300850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.300775    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:52.596551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:52.596524    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"]
Nov 02 15:01:52.597745 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:01:52.597718    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d405b00_cf02_4e4c_ae08_53b0f5ab5980.slice/crio-d7b6f160fc2ca8ed33f9faff0547ef84d8991ea6e361d888b0f8ef60ff586510 WatchSource:0}: Error finding container d7b6f160fc2ca8ed33f9faff0547ef84d8991ea6e361d888b0f8ef60ff586510: Status 404 returned error can't find the container with id d7b6f160fc2ca8ed33f9faff0547ef84d8991ea6e361d888b0f8ef60ff586510
Nov 02 15:01:52.704103 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:01:52.704067    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:01:53.342779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:53.342747    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz" event={"ID":"0d405b00-cf02-4e4c-ae08-53b0f5ab5980","Type":"ContainerStarted","Data":"d7b6f160fc2ca8ed33f9faff0547ef84d8991ea6e361d888b0f8ef60ff586510"}
Nov 02 15:01:58.372721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:58.372680    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz" event={"ID":"0d405b00-cf02-4e4c-ae08-53b0f5ab5980","Type":"ContainerStarted","Data":"55cbef73666433a65ce4d20225e9fea72229bd2852758b74c4822eb9ce0da0b3"}
Nov 02 15:01:58.373204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:58.372791    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:01:58.417767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:01:58.417710    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz" podStartSLOduration=2.203855661 podStartE2EDuration="7.417634598s" totalImagesPullingTime="5.213778937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:01:58.416775758 +0000 UTC m=+312.670729384" watchObservedRunningTime="2026-11-02 15:01:58.417634598 +0000 UTC m=+312.671588222"
Nov 02 15:02:04.407613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:04.407573    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jc2wz"
Nov 02 15:02:05.087147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:05.087094    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:05.236341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:05.236305    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:11.495192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:11.495155    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:11.638306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:11.638266    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:11.785798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:11.785691    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:11.948800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:11.948765    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:15.583664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:15.583622    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:15.762621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:15.762586    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:17.268048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.268014    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"]
Nov 02 15:02:17.270708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.270691    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.290114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.290088    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 15:02:17.302145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.302119    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-cg8jb\""
Nov 02 15:02:17.303579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.303551    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"]
Nov 02 15:02:17.353470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.353439    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 15:02:17.353627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.353520    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 15:02:17.353627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.353584    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 15:02:17.353794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.353780    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 15:02:17.466616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.466573    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9801b861-5043-4810-96b3-e96603bd9e47-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.466828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.466632    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9801b861-5043-4810-96b3-e96603bd9e47-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.466828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.466755    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9801b861-5043-4810-96b3-e96603bd9e47-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.466828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.466813    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwgq4\" (UniqueName: \"kubernetes.io/projected/9801b861-5043-4810-96b3-e96603bd9e47-kube-api-access-xwgq4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.472085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.472058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:17.568178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.568100    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9801b861-5043-4810-96b3-e96603bd9e47-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.568178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.568141    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9801b861-5043-4810-96b3-e96603bd9e47-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.568178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.568161    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9801b861-5043-4810-96b3-e96603bd9e47-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.568394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.568212    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwgq4\" (UniqueName: \"kubernetes.io/projected/9801b861-5043-4810-96b3-e96603bd9e47-kube-api-access-xwgq4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.568812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.568786    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9801b861-5043-4810-96b3-e96603bd9e47-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.569105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.569088    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9801b861-5043-4810-96b3-e96603bd9e47-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.570569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.570548    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9801b861-5043-4810-96b3-e96603bd9e47-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.598491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.598458    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwgq4\" (UniqueName: \"kubernetes.io/projected/9801b861-5043-4810-96b3-e96603bd9e47-kube-api-access-xwgq4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-ngklw\" (UID: \"9801b861-5043-4810-96b3-e96603bd9e47\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:17.662681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.662638    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:17.881783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:17.881754    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:18.182905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:18.182875    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"]
Nov 02 15:02:18.184169 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:02:18.184140    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9801b861_5043_4810_96b3_e96603bd9e47.slice/crio-e9c3ac75f0b76c516148021752676f7efd6c6c480ac72149175a527796557db3 WatchSource:0}: Error finding container e9c3ac75f0b76c516148021752676f7efd6c6c480ac72149175a527796557db3: Status 404 returned error can't find the container with id e9c3ac75f0b76c516148021752676f7efd6c6c480ac72149175a527796557db3
Nov 02 15:02:18.468710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:18.468630    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw" event={"ID":"9801b861-5043-4810-96b3-e96603bd9e47","Type":"ContainerStarted","Data":"e9c3ac75f0b76c516148021752676f7efd6c6c480ac72149175a527796557db3"}
Nov 02 15:02:22.709276 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:02:22.709239    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:02:24.332411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.332371    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:24.490218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.490185    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:24.599154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.599085    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6bwg2"]
Nov 02 15:02:24.663356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.663311    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:24.683390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.683357    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6bwg2"]
Nov 02 15:02:24.683587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.683494    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:24.683587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.683533    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"]
Nov 02 15:02:24.695416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.695388    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"]
Nov 02 15:02:24.695552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.695513    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:24.697005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.696985    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hh45n"]
Nov 02 15:02:24.703262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.703244    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:24.721418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.721345    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hh45n"]
Nov 02 15:02:24.744862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.744830    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4267139-20cc-4984-988b-bbd1082c74b1-tls-certs\") pod \"model-serving-api-75bd89fdfc-hh45n\" (UID: \"d4267139-20cc-4984-988b-bbd1082c74b1\") " pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:24.745021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.744922    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ttdq6\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:24.745021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.744991    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d8e761f-f938-4be7-97c2-bbc0e34045d5-cert\") pod \"kserve-controller-manager-776cc66d47-6bwg2\" (UID: \"5d8e761f-f938-4be7-97c2-bbc0e34045d5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:24.745145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.745103    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7xxw\" (UniqueName: \"kubernetes.io/projected/d4267139-20cc-4984-988b-bbd1082c74b1-kube-api-access-h7xxw\") pod \"model-serving-api-75bd89fdfc-hh45n\" (UID: \"d4267139-20cc-4984-988b-bbd1082c74b1\") " pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:24.745198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.745162    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rqnk\" (UniqueName: \"kubernetes.io/projected/5d8e761f-f938-4be7-97c2-bbc0e34045d5-kube-api-access-7rqnk\") pod \"kserve-controller-manager-776cc66d47-6bwg2\" (UID: \"5d8e761f-f938-4be7-97c2-bbc0e34045d5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:24.745278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.745261    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqn6\" (UniqueName: \"kubernetes.io/projected/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-kube-api-access-khqn6\") pod \"llmisvc-controller-manager-869d5dcc5d-ttdq6\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:24.776373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.776344    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 02 15:02:24.776521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.776420    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 15:02:24.783069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.783044    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8f7br\""
Nov 02 15:02:24.783069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.783042    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-v5xt5\""
Nov 02 15:02:24.783233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.783074    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 15:02:24.783233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.783125    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-w5d6c\""
Nov 02 15:02:24.817065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.817022    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:24.846180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.846144    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ttdq6\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:24.846351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.846231    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d8e761f-f938-4be7-97c2-bbc0e34045d5-cert\") pod \"kserve-controller-manager-776cc66d47-6bwg2\" (UID: \"5d8e761f-f938-4be7-97c2-bbc0e34045d5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:24.846351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.846256    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7xxw\" (UniqueName: \"kubernetes.io/projected/d4267139-20cc-4984-988b-bbd1082c74b1-kube-api-access-h7xxw\") pod \"model-serving-api-75bd89fdfc-hh45n\" (UID: \"d4267139-20cc-4984-988b-bbd1082c74b1\") " pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:24.846351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.846283    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rqnk\" (UniqueName: \"kubernetes.io/projected/5d8e761f-f938-4be7-97c2-bbc0e34045d5-kube-api-access-7rqnk\") pod \"kserve-controller-manager-776cc66d47-6bwg2\" (UID: \"5d8e761f-f938-4be7-97c2-bbc0e34045d5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:24.846351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.846316    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqn6\" (UniqueName: \"kubernetes.io/projected/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-kube-api-access-khqn6\") pod \"llmisvc-controller-manager-869d5dcc5d-ttdq6\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:24.846577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.846364    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4267139-20cc-4984-988b-bbd1082c74b1-tls-certs\") pod \"model-serving-api-75bd89fdfc-hh45n\" (UID: \"d4267139-20cc-4984-988b-bbd1082c74b1\") " pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:24.849307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.849248    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4267139-20cc-4984-988b-bbd1082c74b1-tls-certs\") pod \"model-serving-api-75bd89fdfc-hh45n\" (UID: \"d4267139-20cc-4984-988b-bbd1082c74b1\") " pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:24.849775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.849760    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d8e761f-f938-4be7-97c2-bbc0e34045d5-cert\") pod \"kserve-controller-manager-776cc66d47-6bwg2\" (UID: \"5d8e761f-f938-4be7-97c2-bbc0e34045d5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:24.850246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.850223    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ttdq6\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:24.870543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.870520    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7xxw\" (UniqueName: \"kubernetes.io/projected/d4267139-20cc-4984-988b-bbd1082c74b1-kube-api-access-h7xxw\") pod \"model-serving-api-75bd89fdfc-hh45n\" (UID: \"d4267139-20cc-4984-988b-bbd1082c74b1\") " pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:24.870694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.870598    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqn6\" (UniqueName: \"kubernetes.io/projected/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-kube-api-access-khqn6\") pod \"llmisvc-controller-manager-869d5dcc5d-ttdq6\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:24.870694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:24.870615    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rqnk\" (UniqueName: \"kubernetes.io/projected/5d8e761f-f938-4be7-97c2-bbc0e34045d5-kube-api-access-7rqnk\") pod \"kserve-controller-manager-776cc66d47-6bwg2\" (UID: \"5d8e761f-f938-4be7-97c2-bbc0e34045d5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:25.004861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.004830    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:25.047926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.047705    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:25.068770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.068741    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:25.151165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.151133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:25.340181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.340154    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6bwg2"]
Nov 02 15:02:25.341285 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:02:25.341262    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8e761f_f938_4be7_97c2_bbc0e34045d5.slice/crio-a8902dcb81b23716fa4e53c806c2f3d74b00cfc8ec8132ba769360a130db3604 WatchSource:0}: Error finding container a8902dcb81b23716fa4e53c806c2f3d74b00cfc8ec8132ba769360a130db3604: Status 404 returned error can't find the container with id a8902dcb81b23716fa4e53c806c2f3d74b00cfc8ec8132ba769360a130db3604
Nov 02 15:02:25.378403 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:02:25.378335    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod25cb68c4_8cbb_40ea_912f_dcef1c07d24e.slice/crio-875703daae60bf3ab1f7b98e0d1d44cd791e2460d224dbf927446d10881baf05 WatchSource:0}: Error finding container 875703daae60bf3ab1f7b98e0d1d44cd791e2460d224dbf927446d10881baf05: Status 404 returned error can't find the container with id 875703daae60bf3ab1f7b98e0d1d44cd791e2460d224dbf927446d10881baf05
Nov 02 15:02:25.378642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.378618    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"]
Nov 02 15:02:25.411171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.411146    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hh45n"]
Nov 02 15:02:25.411864 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:02:25.411840    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4267139_20cc_4984_988b_bbd1082c74b1.slice/crio-1a479f437c37c273e29e5f952a97d415252fe4cb73e57c94058c4211e4990a13 WatchSource:0}: Error finding container 1a479f437c37c273e29e5f952a97d415252fe4cb73e57c94058c4211e4990a13: Status 404 returned error can't find the container with id 1a479f437c37c273e29e5f952a97d415252fe4cb73e57c94058c4211e4990a13
Nov 02 15:02:25.520835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.520799    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2" event={"ID":"5d8e761f-f938-4be7-97c2-bbc0e34045d5","Type":"ContainerStarted","Data":"a8902dcb81b23716fa4e53c806c2f3d74b00cfc8ec8132ba769360a130db3604"}
Nov 02 15:02:25.522239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.522205    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hh45n" event={"ID":"d4267139-20cc-4984-988b-bbd1082c74b1","Type":"ContainerStarted","Data":"1a479f437c37c273e29e5f952a97d415252fe4cb73e57c94058c4211e4990a13"}
Nov 02 15:02:25.523606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:25.523577    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6" event={"ID":"25cb68c4-8cbb-40ea-912f-dcef1c07d24e","Type":"ContainerStarted","Data":"875703daae60bf3ab1f7b98e0d1d44cd791e2460d224dbf927446d10881baf05"}
Nov 02 15:02:27.141776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:27.141714    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:27.296017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:27.295982    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:27.439497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:27.439267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:27.663564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:27.663521    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:28.556405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:28.556328    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hh45n" event={"ID":"d4267139-20cc-4984-988b-bbd1082c74b1","Type":"ContainerStarted","Data":"66558cb90180e69e18b6096c43db2474a0c7155d8558b993dd2af4c90be685d2"}
Nov 02 15:02:28.557221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:28.557198    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:28.603119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:28.603058    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-hh45n" podStartSLOduration=1.766112624 podStartE2EDuration="4.603037109s" totalImagesPullingTime="2.836924485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:28.601497819 +0000 UTC m=+342.855451522" watchObservedRunningTime="2026-11-02 15:02:28.603037109 +0000 UTC m=+342.856990734"
Nov 02 15:02:30.574852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.574815    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6" event={"ID":"25cb68c4-8cbb-40ea-912f-dcef1c07d24e","Type":"ContainerStarted","Data":"df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101"}
Nov 02 15:02:30.575232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.574997    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:02:30.576679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.576633    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw" event={"ID":"9801b861-5043-4810-96b3-e96603bd9e47","Type":"ContainerStarted","Data":"5bee307197633334e3633a10c4088c134fa2e6d2246e27443e681b988f3b5904"}
Nov 02 15:02:30.576799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.576709    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:30.578300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.578264    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2" event={"ID":"5d8e761f-f938-4be7-97c2-bbc0e34045d5","Type":"ContainerStarted","Data":"17e28508879f4b0a7d6f2fd8d03b4733a66a55dca7ce90bc8d4e5e9796c04efc"}
Nov 02 15:02:30.578410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.578326    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:02:30.616490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.616435    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6" podStartSLOduration=2.36091104 podStartE2EDuration="6.616420805s" totalImagesPullingTime="4.255509765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:30.615947425 +0000 UTC m=+344.869901050" watchObservedRunningTime="2026-11-02 15:02:30.616420805 +0000 UTC m=+344.870374430"
Nov 02 15:02:30.658361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.658308    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2" podStartSLOduration=2.367879051 podStartE2EDuration="6.658293895s" totalImagesPullingTime="4.290414844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:30.656931572 +0000 UTC m=+344.910885197" watchObservedRunningTime="2026-11-02 15:02:30.658293895 +0000 UTC m=+344.912247520"
Nov 02 15:02:30.698696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:30.698626    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw" podStartSLOduration=1.406258673 podStartE2EDuration="13.698611996s" totalImagesPullingTime="12.292353323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:30.697875127 +0000 UTC m=+344.951828776" watchObservedRunningTime="2026-11-02 15:02:30.698611996 +0000 UTC m=+344.952565611"
Nov 02 15:02:38.079283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:38.079219    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:40.582124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:40.582092    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-hh45n"
Nov 02 15:02:41.588876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:41.588846    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-ngklw"
Nov 02 15:02:42.939447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:42.939411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:47.360575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:47.360538    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:47.504065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:47.504027    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:48.055036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:48.054999    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:48.942208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:48.942144    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:49.090009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:49.089912    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:02:50.958017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:50.957985    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:02:52.715493 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:02:52.715458    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:02:58.765831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:58.765785    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-flxcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:02:58.766325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:58.765873    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:02:59.913271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:02:59.913235    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:00.076098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:00.076052    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:01.585076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.585047    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:03:01.588104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.588083    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6bwg2"
Nov 02 15:03:01.741842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.741809    7019 generic.go:356] "Generic (PLEG): container finished" podID="67e54497-7979-492f-8d4c-6b3741b7d437" containerID="adef7850b2a6e2f390c47d5cd24c9a2893bef01bbd1e6980717a999ac8c6cbcb" exitCode=0
Nov 02 15:03:01.742004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.741867    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" event={"ID":"67e54497-7979-492f-8d4c-6b3741b7d437","Type":"ContainerDied","Data":"adef7850b2a6e2f390c47d5cd24c9a2893bef01bbd1e6980717a999ac8c6cbcb"}
Nov 02 15:03:01.742247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.742226    7019 scope.go:122] "RemoveContainer" containerID="adef7850b2a6e2f390c47d5cd24c9a2893bef01bbd1e6980717a999ac8c6cbcb"
Nov 02 15:03:01.743466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.743437    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:03:01.743522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.743507    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:03:01.743557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:01.743539    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:03:02.749213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:02.749174    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" event={"ID":"67e54497-7979-492f-8d4c-6b3741b7d437","Type":"ContainerStarted","Data":"c44adf0c3429bfe11c3732f0ad88a25903636785f285716aad02a5e219451f08"}
Nov 02 15:03:03.498486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:03.498447    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 15:03:03.498708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:03.498683    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-flxcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body=
Nov 02 15:03:03.498765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:03.498749    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused"
Nov 02 15:03:04.498702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:04.498666    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-flxcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body=
Nov 02 15:03:04.499102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:04.498722    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused"
Nov 02 15:03:05.498420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:05.498383    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-flxcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body=
Nov 02 15:03:05.498601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:05.498437    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused"
Nov 02 15:03:06.498844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:06.498806    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-flxcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body=
Nov 02 15:03:06.499243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:06.498869    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused"
Nov 02 15:03:07.498757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:07.498715    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-flxcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused" start-of-body=
Nov 02 15:03:07.498935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:07.498793    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.23:15021/healthz/ready\": dial tcp 10.134.0.23:15021: connect: connection refused"
Nov 02 15:03:08.501393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:08.501362    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 15:03:08.502010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:08.501989    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 15:03:08.502745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:08.502719    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 15:03:22.704700 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:03:22.704657    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:03:25.686637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.686601    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"]
Nov 02 15:03:25.703080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.703049    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:25.713488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.713461    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"]
Nov 02 15:03:25.769497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.769465    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f0dc694-6e68-4cb1-a298-2be54b98a7b0-cert\") pod \"llmisvc-controller-manager-6466c554d8-lqzvk\" (UID: \"6f0dc694-6e68-4cb1-a298-2be54b98a7b0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:25.769678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.769535    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6fv\" (UniqueName: \"kubernetes.io/projected/6f0dc694-6e68-4cb1-a298-2be54b98a7b0-kube-api-access-hf6fv\") pod \"llmisvc-controller-manager-6466c554d8-lqzvk\" (UID: \"6f0dc694-6e68-4cb1-a298-2be54b98a7b0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:25.870362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.870335    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f0dc694-6e68-4cb1-a298-2be54b98a7b0-cert\") pod \"llmisvc-controller-manager-6466c554d8-lqzvk\" (UID: \"6f0dc694-6e68-4cb1-a298-2be54b98a7b0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:25.870505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.870427    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6fv\" (UniqueName: \"kubernetes.io/projected/6f0dc694-6e68-4cb1-a298-2be54b98a7b0-kube-api-access-hf6fv\") pod \"llmisvc-controller-manager-6466c554d8-lqzvk\" (UID: \"6f0dc694-6e68-4cb1-a298-2be54b98a7b0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:25.873005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.872979    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f0dc694-6e68-4cb1-a298-2be54b98a7b0-cert\") pod \"llmisvc-controller-manager-6466c554d8-lqzvk\" (UID: \"6f0dc694-6e68-4cb1-a298-2be54b98a7b0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:25.896278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:25.896253    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6fv\" (UniqueName: \"kubernetes.io/projected/6f0dc694-6e68-4cb1-a298-2be54b98a7b0-kube-api-access-hf6fv\") pod \"llmisvc-controller-manager-6466c554d8-lqzvk\" (UID: \"6f0dc694-6e68-4cb1-a298-2be54b98a7b0\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:26.017130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:26.017051    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:26.314712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:26.314681    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"]
Nov 02 15:03:26.316247 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:03:26.316222    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0dc694_6e68_4cb1_a298_2be54b98a7b0.slice/crio-67ada1468ad17bfea6237a64a9c26ae0262d571ab6cdc02387a8f91b68f0e5e9 WatchSource:0}: Error finding container 67ada1468ad17bfea6237a64a9c26ae0262d571ab6cdc02387a8f91b68f0e5e9: Status 404 returned error can't find the container with id 67ada1468ad17bfea6237a64a9c26ae0262d571ab6cdc02387a8f91b68f0e5e9
Nov 02 15:03:26.884352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:26.884271    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk" event={"ID":"6f0dc694-6e68-4cb1-a298-2be54b98a7b0","Type":"ContainerStarted","Data":"abf3924d17c5b04b87c101ea6748215af51e08a06b097bd4d6f05c283840ae04"}
Nov 02 15:03:26.884352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:26.884330    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk" event={"ID":"6f0dc694-6e68-4cb1-a298-2be54b98a7b0","Type":"ContainerStarted","Data":"67ada1468ad17bfea6237a64a9c26ae0262d571ab6cdc02387a8f91b68f0e5e9"}
Nov 02 15:03:26.884800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:26.884357    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:26.928517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:26.928465    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk" podStartSLOduration=1.6346004459999999 podStartE2EDuration="1.928451428s" totalImagesPullingTime="293.850982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:03:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:03:26.927629209 +0000 UTC m=+401.181582835" watchObservedRunningTime="2026-11-02 15:03:26.928451428 +0000 UTC m=+401.182405052"
Nov 02 15:03:38.206094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:38.206059    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:43.074052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:43.074019    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:44.388156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:44.388123    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:48.156146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:48.156111    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:52.703876 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:03:52.703828    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:03:57.889299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:57.889263    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-lqzvk"
Nov 02 15:03:57.964580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:57.964545    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"]
Nov 02 15:03:57.964832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:57.964787    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6" podUID="25cb68c4-8cbb-40ea-912f-dcef1c07d24e" containerName="manager" containerID="cri-o://df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101" gracePeriod=30
Nov 02 15:03:58.206092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.206068    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:03:58.255250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.255221    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-cert\" (UniqueName: \"kubernetes.io/secret/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-cert\") pod \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") "
Nov 02 15:03:58.255409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.255339    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-kube-api-access-khqn6\" (UniqueName: \"kubernetes.io/projected/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-kube-api-access-khqn6\") pod \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\" (UID: \"25cb68c4-8cbb-40ea-912f-dcef1c07d24e\") "
Nov 02 15:03:58.257491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.257461    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-cert" pod "25cb68c4-8cbb-40ea-912f-dcef1c07d24e" (UID: "25cb68c4-8cbb-40ea-912f-dcef1c07d24e"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:58.257595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.257508    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-kube-api-access-khqn6" pod "25cb68c4-8cbb-40ea-912f-dcef1c07d24e" (UID: "25cb68c4-8cbb-40ea-912f-dcef1c07d24e"). InnerVolumeSpecName "kube-api-access-khqn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:58.356093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.356060    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqn6\" (UniqueName: \"kubernetes.io/projected/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-kube-api-access-khqn6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:03:58.356093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.356088    7019 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25cb68c4-8cbb-40ea-912f-dcef1c07d24e-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:03:58.796331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.796297    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:58.938891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:58.938854    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:03:59.038960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.038929    7019 generic.go:356] "Generic (PLEG): container finished" podID="25cb68c4-8cbb-40ea-912f-dcef1c07d24e" containerID="df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101" exitCode=0
Nov 02 15:03:59.039121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.038975    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"
Nov 02 15:03:59.039121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.039013    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6" event={"ID":"25cb68c4-8cbb-40ea-912f-dcef1c07d24e","Type":"ContainerDied","Data":"df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101"}
Nov 02 15:03:59.039121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.039060    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6" event={"ID":"25cb68c4-8cbb-40ea-912f-dcef1c07d24e","Type":"ContainerDied","Data":"875703daae60bf3ab1f7b98e0d1d44cd791e2460d224dbf927446d10881baf05"}
Nov 02 15:03:59.039121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.039082    7019 scope.go:122] "RemoveContainer" containerID="df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101"
Nov 02 15:03:59.048106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.048090    7019 scope.go:122] "RemoveContainer" containerID="df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101"
Nov 02 15:03:59.048384 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:03:59.048365    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101\": container with ID starting with df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101 not found: ID does not exist" containerID="df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101"
Nov 02 15:03:59.048432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.048394    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101"} err="failed to get container status \"df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101\": rpc error: code = NotFound desc = could not find container \"df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101\": container with ID starting with df773474d12326e0d094eb6b790e918e30bcaba849841d84269ed490a79c1101 not found: ID does not exist"
Nov 02 15:03:59.094362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.094336    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"]
Nov 02 15:03:59.109662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.109619    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ttdq6"]
Nov 02 15:03:59.860687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:03:59.860629    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cb68c4-8cbb-40ea-912f-dcef1c07d24e" path="/var/lib/kubelet/pods/25cb68c4-8cbb-40ea-912f-dcef1c07d24e/volumes"
Nov 02 15:04:17.236764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:17.236722    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:18.272222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:18.272182    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:04:19.464567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:19.464535    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:21.889281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:21.889241    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:22.704903 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:04:22.704865    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:04:23.169220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:23.169178    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:04:23.876133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:23.876097    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:26.030041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:26.030007    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:27.518823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:27.518743    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:04:27.722409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:27.722369    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:29.354875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:29.354831    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:31.206938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:31.206900    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:34.836267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:34.836225    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:04:38.323102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:38.323058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:43.184844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:43.184800    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:45.504876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:45.504831    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:45.686608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:45.686581    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:48.277606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:48.277560    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:50.673896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:50.673855    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:50.823075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:50.823031    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:04:52.704121 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:04:52.704078    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:04:55.862124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:04:55.862090    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:01.987166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:01.987124    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:02.144394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:02.144358    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:06.153132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.153103    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"]
Nov 02 15:05:06.153569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.153555    7019 cpu_manager.go:414] "containerMap: removing container" podUID="25cb68c4-8cbb-40ea-912f-dcef1c07d24e" containerName="manager"
Nov 02 15:05:06.153618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.153570    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cb68c4-8cbb-40ea-912f-dcef1c07d24e" containerName="manager"
Nov 02 15:05:06.153676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.153636    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cb68c4-8cbb-40ea-912f-dcef1c07d24e" containerName="manager"
Nov 02 15:05:06.156093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.156073    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.184494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.184461    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"]
Nov 02 15:05:06.225587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.225554    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00052150-74c8-4c65-8395-6b888102714b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.225749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.225592    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.225749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.225616    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.225749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.225695    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.225854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.225810    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.225886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.225851    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00052150-74c8-4c65-8395-6b888102714b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.225955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.225937    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.226022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.226009    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf47d\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-kube-api-access-pf47d\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.226055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.226036    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327264    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf47d\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-kube-api-access-pf47d\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327305    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327330    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00052150-74c8-4c65-8395-6b888102714b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327456    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327498    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327520    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327556    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327612    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00052150-74c8-4c65-8395-6b888102714b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327718    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327759    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.327989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.327858    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.328126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.328108    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.328188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.328108    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00052150-74c8-4c65-8395-6b888102714b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.328188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.328177    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.329965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.329947    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.330256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.330236    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00052150-74c8-4c65-8395-6b888102714b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.351175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.351145    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf47d\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-kube-api-access-pf47d\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.351279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.351189    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-tvp64\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.468839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.468746    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:06.779014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.778989    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"]
Nov 02 15:05:06.781222 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:05:06.781196    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00052150_74c8_4c65_8395_6b888102714b.slice/crio-ccade901c9ae3b84070aa47e061e40d8a03e63f070dc9b02b1301e5866eee6cb WatchSource:0}: Error finding container ccade901c9ae3b84070aa47e061e40d8a03e63f070dc9b02b1301e5866eee6cb: Status 404 returned error can't find the container with id ccade901c9ae3b84070aa47e061e40d8a03e63f070dc9b02b1301e5866eee6cb
Nov 02 15:05:06.783345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.783308    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:05:06.783423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.783381    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:05:06.783423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:06.783411    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:05:07.395341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:07.395303    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" event={"ID":"00052150-74c8-4c65-8395-6b888102714b","Type":"ContainerStarted","Data":"a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7"}
Nov 02 15:05:07.395806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:07.395357    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" event={"ID":"00052150-74c8-4c65-8395-6b888102714b","Type":"ContainerStarted","Data":"ccade901c9ae3b84070aa47e061e40d8a03e63f070dc9b02b1301e5866eee6cb"}
Nov 02 15:05:07.484670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:07.479255    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podStartSLOduration=1.479233742 podStartE2EDuration="1.479233742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:07.453237635 +0000 UTC m=+501.707191258" watchObservedRunningTime="2026-11-02 15:05:07.479233742 +0000 UTC m=+501.733187369"
Nov 02 15:05:07.489985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:07.489892    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:07.491221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:07.491195    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 15:05:07.491336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:07.491245    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 15:05:08.483130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:08.483090    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 15:05:08.483784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:08.483742    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 15:05:09.487108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:09.487071    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 15:05:09.487753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:09.487719    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 15:05:10.478057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:10.478019    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 15:05:10.478326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:10.478298    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 15:05:11.484667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:11.484011    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 15:05:11.484667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:11.484073    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 15:05:12.485673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:12.471018    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 15:05:12.485673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:12.471081    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 15:05:13.469922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:13.469890    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 15:05:13.470118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:13.469950    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 15:05:14.915597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:14.915537    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:14.916120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:14.915994    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:14.916858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:14.916836    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:05:15.560627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:15.560594    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:16.753347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:16.753310    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:17.275625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:17.275584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:18.243114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:18.243075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:18.394434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:18.394399    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:20.120127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:20.120092    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:20.476760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:20.476678    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:20.643347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:20.643312    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:22.703584 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:05:22.703551    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:05:34.152264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:34.152210    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:36.489294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:36.489256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:38.484424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:38.484379    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:43.308238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:43.308199    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:45.384808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:45.384775    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:45.529911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:45.529882    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:48.390270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:48.390232    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:52.268942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:52.268895    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:52.405315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:52.405279    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:52.705833 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:05:52.705795    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:05:52.875390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:52.875353    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:53.019091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:53.018986    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:53.170629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:53.170588    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:53.352392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:53.352303    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:53.523978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:53.523926    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:53.676387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:53.676348    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:53.827614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:53.827576    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:56.910132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:56.910030    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:57.080102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:57.080067    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:57.241484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:57.241391    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:57.519085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:57.519005    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:57.666580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:57.666543    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:57.818993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:57.818905    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:57.958699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:57.958664    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:58.115264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:58.115234    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:58.760532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:58.760500    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:05:58.953514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:05:58.953474    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:00.178896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:00.178846    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:00.336160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:00.336127    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:00.485315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:00.485225    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:01.340382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:01.340324    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:01.501476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:01.501441    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:02.353144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.353104    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:02.499202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.499167    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:02.688665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.688623    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"]
Nov 02 15:06:02.693038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.693015    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.709463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.709434    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5113c2-openshift-default-dockercfg-kwr29\""
Nov 02 15:06:02.723106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.723077    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"]
Nov 02 15:06:02.783282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.783245    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:02.794700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.794662    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.794870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.794748    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.794920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.794867    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5ddd8-f388-4762-9fd9-ba1231de0867-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.794955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.794945    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.795032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.794977    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mb8\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-kube-api-access-c5mb8\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.795032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.794999    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.795140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.795028    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.795213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.795188    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.795275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.795241    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896115    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896161    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896197    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896227    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896271    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896296    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5ddd8-f388-4762-9fd9-ba1231de0867-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896369    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896406    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mb8\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-kube-api-access-c5mb8\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896437    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896637    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896678    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.896969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.896945    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.897172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.897147    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5ddd8-f388-4762-9fd9-ba1231de0867-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.897344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.897330    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.898895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.898866    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.899061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.899045    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.923114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.923084    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.928337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.928302    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mb8\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-kube-api-access-c5mb8\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:02.942799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:02.942708    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:03.008064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.008026    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:03.090261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.090191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:03.329864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.329838    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"]
Nov 02 15:06:03.331540 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:06:03.331505    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec5ddd8_f388_4762_9fd9_ba1231de0867.slice/crio-29fb004f8ee33a9f9dbc58887a8f6a2bd7f9e864799fd556f42b2c3b3381ef0d WatchSource:0}: Error finding container 29fb004f8ee33a9f9dbc58887a8f6a2bd7f9e864799fd556f42b2c3b3381ef0d: Status 404 returned error can't find the container with id 29fb004f8ee33a9f9dbc58887a8f6a2bd7f9e864799fd556f42b2c3b3381ef0d
Nov 02 15:06:03.333834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.333799    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:03.333944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.333887    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:03.334016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.333998    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:03.685180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.685141    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" event={"ID":"dec5ddd8-f388-4762-9fd9-ba1231de0867","Type":"ContainerStarted","Data":"73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60"}
Nov 02 15:06:03.685592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.685212    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" event={"ID":"dec5ddd8-f388-4762-9fd9-ba1231de0867","Type":"ContainerStarted","Data":"29fb004f8ee33a9f9dbc58887a8f6a2bd7f9e864799fd556f42b2c3b3381ef0d"}
Nov 02 15:06:03.756397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.756343    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podStartSLOduration=1.7563259690000002 podStartE2EDuration="1.756325969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:03.743026182 +0000 UTC m=+557.996979809" watchObservedRunningTime="2026-11-02 15:06:03.756325969 +0000 UTC m=+558.010279590"
Nov 02 15:06:03.807816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.807770    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:03.963190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:03.963100    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:04.009466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.009435    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:04.015601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.015574    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:04.155036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.155004    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:06:04.181129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.181098    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:04.338686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.338566    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:04.486838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.486807    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:04.692387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.692351    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:04.693478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:04.693455    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:06.073102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:06.073062    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:06.369905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:06.369865    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:06.585693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:06.585657    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:06.810265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:06.810170    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:06.957514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:06.957476    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:08.199470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:08.199420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:08.644833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:08.644793    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:08.823598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:08.823558    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:09.713021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.712985    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:09.877385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.877348    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:09.935490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.935452    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"]
Nov 02 15:06:09.938612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.938585    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:09.959451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.959424    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"]
Nov 02 15:06:09.970820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.970717    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c2a0e-d79c-4f85-bd85-2375c58ce916-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:09.970981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.970896    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvrg\" (UniqueName: \"kubernetes.io/projected/729c2a0e-d79c-4f85-bd85-2375c58ce916-kube-api-access-2jvrg\") pod \"payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:09.976002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.975960    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-7ecfe1-dockercfg-lhw7s\""
Nov 02 15:06:09.976471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:09.976453    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-7ecfe1\""
Nov 02 15:06:10.072069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.072030    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c2a0e-d79c-4f85-bd85-2375c58ce916-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:10.072237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.072191    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvrg\" (UniqueName: \"kubernetes.io/projected/729c2a0e-d79c-4f85-bd85-2375c58ce916-kube-api-access-2jvrg\") pod \"payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:10.073020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.072991    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c2a0e-d79c-4f85-bd85-2375c58ce916-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:10.096479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.096447    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvrg\" (UniqueName: \"kubernetes.io/projected/729c2a0e-d79c-4f85-bd85-2375c58ce916-kube-api-access-2jvrg\") pod \"payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:10.251689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.251598    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:10.366701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.366636    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:10.471682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.471627    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:06:10.506314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.506220    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:10.557877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.557803    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"]
Nov 02 15:06:10.559939 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:06:10.559906    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729c2a0e_d79c_4f85_bd85_2375c58ce916.slice/crio-a4944fb8be5165b343f42b722ed9ffb91b3af257d51682a407310c77b3d3e4f3 WatchSource:0}: Error finding container a4944fb8be5165b343f42b722ed9ffb91b3af257d51682a407310c77b3d3e4f3: Status 404 returned error can't find the container with id a4944fb8be5165b343f42b722ed9ffb91b3af257d51682a407310c77b3d3e4f3
Nov 02 15:06:10.562594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.562568    7019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:06:10.727137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:10.727100    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" event={"ID":"729c2a0e-d79c-4f85-bd85-2375c58ce916","Type":"ContainerStarted","Data":"a4944fb8be5165b343f42b722ed9ffb91b3af257d51682a407310c77b3d3e4f3"}
Nov 02 15:06:11.778558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:11.778409    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:12.067302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.067205    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"]
Nov 02 15:06:12.074145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.074110    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.103684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.101389    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"]
Nov 02 15:06:12.103684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.101783    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-739bf1-serving-cert\""
Nov 02 15:06:12.103684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.101865    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 15:06:12.103684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.101902    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cz4f2\""
Nov 02 15:06:12.187298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.187184    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:12.195333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.195296    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.195524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.195431    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgpc\" (UniqueName: \"kubernetes.io/projected/585b1556-e70a-4d65-905e-9d0bb0555eb2-kube-api-access-htgpc\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.195524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.195488    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585b1556-e70a-4d65-905e-9d0bb0555eb2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.195669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.195613    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-metrics-certs\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.297524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.297482    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.297737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.297597    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgpc\" (UniqueName: \"kubernetes.io/projected/585b1556-e70a-4d65-905e-9d0bb0555eb2-kube-api-access-htgpc\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.297737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.297690    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585b1556-e70a-4d65-905e-9d0bb0555eb2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.297874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.297802    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-metrics-certs\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.298748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.298719    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585b1556-e70a-4d65-905e-9d0bb0555eb2-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.301246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.301219    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-maas-api-tls\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.304403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.304375    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-metrics-certs\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.327880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.327787    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgpc\" (UniqueName: \"kubernetes.io/projected/585b1556-e70a-4d65-905e-9d0bb0555eb2-kube-api-access-htgpc\") pod \"maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.350638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.350533    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:12.402361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.402317    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:12.504373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.504340    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:12.665245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.665206    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:12.723221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.723191    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"]
Nov 02 15:06:12.727749 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:06:12.727723    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585b1556_e70a_4d65_905e_9d0bb0555eb2.slice/crio-15482fc6805649be1c92cb3832849ed2f3ae57d1f6654465dbc13ecd0656c310 WatchSource:0}: Error finding container 15482fc6805649be1c92cb3832849ed2f3ae57d1f6654465dbc13ecd0656c310: Status 404 returned error can't find the container with id 15482fc6805649be1c92cb3832849ed2f3ae57d1f6654465dbc13ecd0656c310
Nov 02 15:06:12.739677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:12.739623    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb" event={"ID":"585b1556-e70a-4d65-905e-9d0bb0555eb2","Type":"ContainerStarted","Data":"15482fc6805649be1c92cb3832849ed2f3ae57d1f6654465dbc13ecd0656c310"}
Nov 02 15:06:13.751519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:13.751478    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" event={"ID":"729c2a0e-d79c-4f85-bd85-2375c58ce916","Type":"ContainerStarted","Data":"19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182"}
Nov 02 15:06:13.752018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:13.751828    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:13.795867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:13.795799    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" podStartSLOduration=2.648848514 podStartE2EDuration="4.79577638s" totalImagesPullingTime="2.146927866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:13.793630713 +0000 UTC m=+568.047584338" watchObservedRunningTime="2026-11-02 15:06:13.79577638 +0000 UTC m=+568.049730006"
Nov 02 15:06:14.069119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:14.069047    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:15.764406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:15.764368    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb" event={"ID":"585b1556-e70a-4d65-905e-9d0bb0555eb2","Type":"ContainerStarted","Data":"736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8"}
Nov 02 15:06:15.764849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:15.764448    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:15.807507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:15.807445    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb" podStartSLOduration=1.24033886 podStartE2EDuration="3.807428006s" totalImagesPullingTime="2.567089146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:15.806764631 +0000 UTC m=+570.060718259" watchObservedRunningTime="2026-11-02 15:06:15.807428006 +0000 UTC m=+570.061381631"
Nov 02 15:06:15.810047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:15.810020    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:16.484345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.484310    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"]
Nov 02 15:06:16.488992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.488970    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.518484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.518445    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"openshift-service-ca.crt\""
Nov 02 15:06:16.518701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.518555    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"kube-root-ca.crt\""
Nov 02 15:06:16.530844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.530812    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"default-dockercfg-l4n54\""
Nov 02 15:06:16.531034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.531011    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-1cc1b2\"/\"simulator-w5-1cc1b2-kserve-self-signed-certs\""
Nov 02 15:06:16.533557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.533532    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"]
Nov 02 15:06:16.547364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.547327    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-dshm\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.547364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.547362    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-model-cache\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.547709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.547402    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-home\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.547709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.547464    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-tmp-dir\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.547709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.547564    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d73acb-b8ac-4620-a00e-0b3a5a719560-tls-certs\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.547709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.547684    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs9lx\" (UniqueName: \"kubernetes.io/projected/95d73acb-b8ac-4620-a00e-0b3a5a719560-kube-api-access-gs9lx\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.591720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.591681    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:16.648532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648495    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-home\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.648532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648532    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-tmp-dir\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.648820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648573    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d73acb-b8ac-4620-a00e-0b3a5a719560-tls-certs\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.648820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648622    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs9lx\" (UniqueName: \"kubernetes.io/projected/95d73acb-b8ac-4620-a00e-0b3a5a719560-kube-api-access-gs9lx\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.648820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648707    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-dshm\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.648820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648740    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-model-cache\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.649037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648929    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-home\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.649037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.648991    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-tmp-dir\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.649145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.649039    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-model-cache\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.651090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.651062    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-dshm\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.651222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.651171    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d73acb-b8ac-4620-a00e-0b3a5a719560-tls-certs\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.673179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.673147    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs9lx\" (UniqueName: \"kubernetes.io/projected/95d73acb-b8ac-4620-a00e-0b3a5a719560-kube-api-access-gs9lx\") pod \"simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") " pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.822734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.822628    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:16.956546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:16.956489    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:17.142264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.142230    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"]
Nov 02 15:06:17.529167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.529078    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"]
Nov 02 15:06:17.533426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.533396    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.559403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.559370    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-5113c2-serving-cert\""
Nov 02 15:06:17.561517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.561491    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"]
Nov 02 15:06:17.583857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.583823    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:17.661688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.661554    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.661822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.661742    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d9ead9-55e5-46cb-85d9-e837fcdf8716-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.661822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.661790    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-metrics-certs\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.662003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.661979    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjmh4\" (UniqueName: \"kubernetes.io/projected/65d9ead9-55e5-46cb-85d9-e837fcdf8716-kube-api-access-hjmh4\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.736124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.736085    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:17.764226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.762743    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.764226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.762813    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d9ead9-55e5-46cb-85d9-e837fcdf8716-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.764226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.762845    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-metrics-certs\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.764226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.762955    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjmh4\" (UniqueName: \"kubernetes.io/projected/65d9ead9-55e5-46cb-85d9-e837fcdf8716-kube-api-access-hjmh4\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.764677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.764607    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d9ead9-55e5-46cb-85d9-e837fcdf8716-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.769137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.769108    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-maas-api-tls\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.775716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.775657    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-metrics-certs\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.787212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.786199    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" event={"ID":"95d73acb-b8ac-4620-a00e-0b3a5a719560","Type":"ContainerStarted","Data":"1b280fa60af0e124f0834ac2534f5811661c26f181cd0310e12353b20562548e"}
Nov 02 15:06:17.789727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.789696    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjmh4\" (UniqueName: \"kubernetes.io/projected/65d9ead9-55e5-46cb-85d9-e837fcdf8716-kube-api-access-hjmh4\") pod \"maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.862719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.862600    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:17.887605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:17.887570    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:18.203780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.203751    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"]
Nov 02 15:06:18.204800 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:06:18.204760    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d9ead9_55e5_46cb_85d9_e837fcdf8716.slice/crio-9d38c7ea4bf304f9ddcbe5dfcb168ff974aa9df49a9d9ebb1ff69e08ce1ceb9c WatchSource:0}: Error finding container 9d38c7ea4bf304f9ddcbe5dfcb168ff974aa9df49a9d9ebb1ff69e08ce1ceb9c: Status 404 returned error can't find the container with id 9d38c7ea4bf304f9ddcbe5dfcb168ff974aa9df49a9d9ebb1ff69e08ce1ceb9c
Nov 02 15:06:18.793669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.793551    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" event={"ID":"95d73acb-b8ac-4620-a00e-0b3a5a719560","Type":"ContainerStarted","Data":"42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16"}
Nov 02 15:06:18.795884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.795845    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs" event={"ID":"65d9ead9-55e5-46cb-85d9-e837fcdf8716","Type":"ContainerStarted","Data":"a81adde0c44e7dfcafc6fcd45c5898d1d7dff8ba55d408c89467996c13b4900e"}
Nov 02 15:06:18.795884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.795908    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs" event={"ID":"65d9ead9-55e5-46cb-85d9-e837fcdf8716","Type":"ContainerStarted","Data":"9d38c7ea4bf304f9ddcbe5dfcb168ff974aa9df49a9d9ebb1ff69e08ce1ceb9c"}
Nov 02 15:06:18.796334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.796311    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:18.823221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.823187    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:18.839092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.839016    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podStartSLOduration=1.549467929 podStartE2EDuration="2.838971153s" totalImagesPullingTime="1.289503224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:18.838846136 +0000 UTC m=+573.092799796" watchObservedRunningTime="2026-11-02 15:06:18.838971153 +0000 UTC m=+573.092924779"
Nov 02 15:06:18.885297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:18.885245    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs" podStartSLOduration=1.557836688 podStartE2EDuration="1.885230272s" totalImagesPullingTime="327.393584ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:18.884972062 +0000 UTC m=+573.138925686" watchObservedRunningTime="2026-11-02 15:06:18.885230272 +0000 UTC m=+573.139183898"
Nov 02 15:06:19.439094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.439057    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"]
Nov 02 15:06:19.443148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.443105    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.457944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.457915    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-7ecfe1-serving-cert\""
Nov 02 15:06:19.475560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.475517    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"]
Nov 02 15:06:19.566365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.566324    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:19.583531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.583490    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c98ff2-f44f-4453-bff2-eaa67af0869c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.583724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.583548    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9pr\" (UniqueName: \"kubernetes.io/projected/10c98ff2-f44f-4453-bff2-eaa67af0869c-kube-api-access-gz9pr\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.583724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.583679    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-metrics-certs\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.583796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.583729    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.684757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.684722    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c98ff2-f44f-4453-bff2-eaa67af0869c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.684945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.684774    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9pr\" (UniqueName: \"kubernetes.io/projected/10c98ff2-f44f-4453-bff2-eaa67af0869c-kube-api-access-gz9pr\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.684945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.684805    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-metrics-certs\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.684945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.684838    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.685505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.685482    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c98ff2-f44f-4453-bff2-eaa67af0869c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.687319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.687294    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-metrics-certs\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.687414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.687335    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.708787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.708700    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:19.709334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.709309    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9pr\" (UniqueName: \"kubernetes.io/projected/10c98ff2-f44f-4453-bff2-eaa67af0869c-kube-api-access-gz9pr\") pod \"maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:19.757962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:19.757926    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:20.060730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:20.060704    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"]
Nov 02 15:06:20.061066 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:06:20.060966    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c98ff2_f44f_4453_bff2_eaa67af0869c.slice/crio-963af7d03c35d7c32e85dea66ddd2f7255c99bac2e7ae91eb52e997d471fc5e6 WatchSource:0}: Error finding container 963af7d03c35d7c32e85dea66ddd2f7255c99bac2e7ae91eb52e997d471fc5e6: Status 404 returned error can't find the container with id 963af7d03c35d7c32e85dea66ddd2f7255c99bac2e7ae91eb52e997d471fc5e6
Nov 02 15:06:20.815193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:20.815091    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6" event={"ID":"10c98ff2-f44f-4453-bff2-eaa67af0869c","Type":"ContainerStarted","Data":"50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305"}
Nov 02 15:06:20.815193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:20.815176    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6" event={"ID":"10c98ff2-f44f-4453-bff2-eaa67af0869c","Type":"ContainerStarted","Data":"963af7d03c35d7c32e85dea66ddd2f7255c99bac2e7ae91eb52e997d471fc5e6"}
Nov 02 15:06:20.815430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:20.815306    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:20.859443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:20.859389    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6" podStartSLOduration=1.527024938 podStartE2EDuration="1.859374644s" totalImagesPullingTime="332.349706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:20.857717362 +0000 UTC m=+575.111670987" watchObservedRunningTime="2026-11-02 15:06:20.859374644 +0000 UTC m=+575.113328265"
Nov 02 15:06:21.385475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:21.385429    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:21.777007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:21.776917    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:06:22.061157 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:22.061075    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:06:22.081003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:22.080960    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:22.702452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:22.702411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:22.715871 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:06:22.715841    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:06:22.853850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:22.853813    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:23.020970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:23.020874    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:23.164828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:23.164790    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:24.761581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:24.761550    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:06:24.808832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:24.808806    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:06:24.902416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:24.902370    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:26.558839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:26.558798    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:26.823240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:26.823151    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:26.826120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:26.826098    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:06:26.835597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:26.835575    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:26.898328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:26.898303    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:06:27.978970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:27.978934    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:28.216755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:28.216715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:28.375211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:28.375168    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:28.804875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:28.804779    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:28.991696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:28.991630    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:30.296428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:30.296392    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:30.829210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:30.829044    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:31.062609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.062579    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:06:31.920577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.920544    7019 generic.go:356] "Generic (PLEG): container finished" podID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerID="73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60" exitCode=0
Nov 02 15:06:31.921027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.920583    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" event={"ID":"dec5ddd8-f388-4762-9fd9-ba1231de0867","Type":"ContainerDied","Data":"73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60"}
Nov 02 15:06:31.921027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.920985    7019 scope.go:122] "RemoveContainer" containerID="73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60"
Nov 02 15:06:31.922458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.922429    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:31.922518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.922500    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:31.922555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.922533    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:06:31.999482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:31.999453    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:32.760736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:32.760703    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:32.928667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:32.928616    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" event={"ID":"dec5ddd8-f388-4762-9fd9-ba1231de0867","Type":"ContainerStarted","Data":"01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730"}
Nov 02 15:06:33.008418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:33.008379    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:33.009368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:33.009340    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 15:06:33.009493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:33.009387    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 15:06:33.437275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:33.437190    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:33.627398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:33.627362    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:33.791352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:33.791253    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:34.008994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:34.008957    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 15:06:34.009510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:34.009058    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 15:06:34.693174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:34.693141    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:35.009054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:35.008963    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 15:06:35.009054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:35.009024    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 15:06:35.585673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:35.582159    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:36.008979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:36.008935    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 15:06:36.009169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:36.008999    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 15:06:36.602055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:36.602018    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:37.008910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:37.008877    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 15:06:37.009082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:37.008933    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 15:06:37.090905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:37.090865    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:37.231154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:37.231118    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:37.397068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:37.394374    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:37.789476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:37.789386    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:37.959369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:37.959328    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:38.008457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:38.008420    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 15:06:38.008643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:38.008483    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 15:06:38.618511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:38.618476    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:39.014012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:39.013923    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:39.015236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:39.015212    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:06:39.068367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:39.067945    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:40.714538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:40.714499    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:41.210850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.210809    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"]
Nov 02 15:06:41.219993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.219959    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.233816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.233783    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"]
Nov 02 15:06:41.243903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.243877    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"openshift-service-ca.crt\""
Nov 02 15:06:41.244057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.243944    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"simulator-w0-6af50f-kserve-self-signed-certs\""
Nov 02 15:06:41.244057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.243971    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"kube-root-ca.crt\""
Nov 02 15:06:41.247035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.247019    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"default-dockercfg-thbcx\""
Nov 02 15:06:41.275659 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.275615    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-dshm\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.275857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.275671    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-model-cache\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.275857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.275701    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/832acbfc-6909-41ec-9aed-b653a7d9a662-kube-api-access-m9mvt\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.275970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.275859    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832acbfc-6909-41ec-9aed-b653a7d9a662-tls-certs\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.276045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.276026    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-tmp-dir\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.276085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.276056    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-home\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.304711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.304667    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:41.377270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377227    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-tmp-dir\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377281    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-home\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377350    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-dshm\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377388    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-model-cache\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377415    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/832acbfc-6909-41ec-9aed-b653a7d9a662-kube-api-access-m9mvt\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377497    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832acbfc-6909-41ec-9aed-b653a7d9a662-tls-certs\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377782    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-tmp-dir\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377887    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-home\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.377975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.377918    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-model-cache\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.379728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.379691    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-dshm\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.380301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.380282    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832acbfc-6909-41ec-9aed-b653a7d9a662-tls-certs\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.401095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.401068    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/832acbfc-6909-41ec-9aed-b653a7d9a662-kube-api-access-m9mvt\") pod \"simulator-w0-6af50f-kserve-78f797bdc7-qmf2p\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") " pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.532371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.532284    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:41.836310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.836281    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"]
Nov 02 15:06:41.836850 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:06:41.836824    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod832acbfc_6909_41ec_9aed_b653a7d9a662.slice/crio-a7a20c633993d95a07a0ed39a547e95d65c12e6ebc6ad918fd5df059eb50e0f9 WatchSource:0}: Error finding container a7a20c633993d95a07a0ed39a547e95d65c12e6ebc6ad918fd5df059eb50e0f9: Status 404 returned error can't find the container with id a7a20c633993d95a07a0ed39a547e95d65c12e6ebc6ad918fd5df059eb50e0f9
Nov 02 15:06:41.991681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.991630    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" event={"ID":"832acbfc-6909-41ec-9aed-b653a7d9a662","Type":"ContainerStarted","Data":"a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00"}
Nov 02 15:06:41.991867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:41.991688    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" event={"ID":"832acbfc-6909-41ec-9aed-b653a7d9a662","Type":"ContainerStarted","Data":"a7a20c633993d95a07a0ed39a547e95d65c12e6ebc6ad918fd5df059eb50e0f9"}
Nov 02 15:06:42.034332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:42.034265    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podStartSLOduration=1.0342468949999999 podStartE2EDuration="1.034246895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:42.033076654 +0000 UTC m=+596.287030282" watchObservedRunningTime="2026-11-02 15:06:42.034246895 +0000 UTC m=+596.288200524"
Nov 02 15:06:42.406887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:42.406846    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:42.532722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:42.532678    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:43.436582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:43.436551    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:43.998337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:43.998275    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:44.652066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:44.652021    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:45.628407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:45.628371    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:45.728462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:45.728430    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:06:47.295453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:47.295419    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:48.540846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:48.540807    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:49.019392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:49.019357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:50.701780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:50.701728    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:50.859324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:50.859285    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:51.533044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:51.532994    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:51.546286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:51.546257    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:52.065535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:52.065497    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:06:52.497741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:52.497707    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:52.706126 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:06:52.706092    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:06:54.099946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:54.099910    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:55.697842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:55.697802    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:55.786610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:55.786577    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:06:57.004037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.003998    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"]
Nov 02 15:06:57.008315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.008291    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.028494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.028470    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"]
Nov 02 15:06:57.038899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.038872    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"openshift-service-ca.crt\""
Nov 02 15:06:57.038899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.038891    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"kube-root-ca.crt\""
Nov 02 15:06:57.039149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.039133    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"default-dockercfg-2cv7c\""
Nov 02 15:06:57.039192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.039140    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"simulator-w4-7ecfe1-kserve-self-signed-certs\""
Nov 02 15:06:57.078609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.078568    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:57.149453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.149410    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a729fd-8c15-46ab-81be-21f5052b2bce-tls-certs\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.149665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.149495    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-home\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.149665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.149579    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-dshm\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.149777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.149678    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-model-cache\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.149777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.149728    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6prz\" (UniqueName: \"kubernetes.io/projected/60a729fd-8c15-46ab-81be-21f5052b2bce-kube-api-access-l6prz\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.149880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.149824    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-tmp-dir\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251337    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a729fd-8c15-46ab-81be-21f5052b2bce-tls-certs\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251388    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-home\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251485    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-dshm\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251511    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-model-cache\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251547    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6prz\" (UniqueName: \"kubernetes.io/projected/60a729fd-8c15-46ab-81be-21f5052b2bce-kube-api-access-l6prz\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251694    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-tmp-dir\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251877    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-home\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.251955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.251940    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-model-cache\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.252124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.252092    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-tmp-dir\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.253923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.253899    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-dshm\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.254097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.254051    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a729fd-8c15-46ab-81be-21f5052b2bce-tls-certs\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.275292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.275262    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6prz\" (UniqueName: \"kubernetes.io/projected/60a729fd-8c15-46ab-81be-21f5052b2bce-kube-api-access-l6prz\") pod \"simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") " pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.320950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.320912    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:57.397613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.397566    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:57.623591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:57.623563    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"]
Nov 02 15:06:57.624688 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:06:57.624665    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a729fd_8c15_46ab_81be_21f5052b2bce.slice/crio-ae8d35bc68d2b857de71ba1e133cde47c65dc7930ee1d7f8a5b3d0d577944cec WatchSource:0}: Error finding container ae8d35bc68d2b857de71ba1e133cde47c65dc7930ee1d7f8a5b3d0d577944cec: Status 404 returned error can't find the container with id ae8d35bc68d2b857de71ba1e133cde47c65dc7930ee1d7f8a5b3d0d577944cec
Nov 02 15:06:58.092864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:58.092771    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" event={"ID":"60a729fd-8c15-46ab-81be-21f5052b2bce","Type":"ContainerStarted","Data":"853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8"}
Nov 02 15:06:58.092864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:58.092817    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" event={"ID":"60a729fd-8c15-46ab-81be-21f5052b2bce","Type":"ContainerStarted","Data":"ae8d35bc68d2b857de71ba1e133cde47c65dc7930ee1d7f8a5b3d0d577944cec"}
Nov 02 15:06:58.136446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:58.136384    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podStartSLOduration=2.1363690650000002 podStartE2EDuration="2.136369065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:58.135399837 +0000 UTC m=+612.389353486" watchObservedRunningTime="2026-11-02 15:06:58.136369065 +0000 UTC m=+612.390322686"
Nov 02 15:06:58.321312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:58.321273    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:06:59.004023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:59.003988    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:06:59.897364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:06:59.897316    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:00.593135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:00.593091    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:02.202858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:02.202817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:03.798848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:03.796839    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:05.484315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:05.484265    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:06.484173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:06.484130    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:07.197482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:07.197444    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:07.321603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:07.321568    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:07:07.334139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:07.334113    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:07:08.189243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:08.189211    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:07:08.797269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:08.797224    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:10.388210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:10.388164    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:11.256752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:11.256720    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:07:11.992088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:11.992045    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:12.733704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:12.733637    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:13.648146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:13.648113    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:15.413970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:15.413924    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:17.089866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:17.089817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:18.706154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:18.706112    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:20.323153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:20.323117    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:21.891243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:21.891203    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:22.708107 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:07:22.708069    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:07:22.993750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:22.991464    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"]
Nov 02 15:07:22.996015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:22.995990    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.016123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.016093    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"]
Nov 02 15:07:23.025892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.025859    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"premium-w0-6af50f-kserve-self-signed-certs\""
Nov 02 15:07:23.046806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.046768    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-home\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.046979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.046822    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-dshm\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.046979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.046948    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stlhp\" (UniqueName: \"kubernetes.io/projected/60cb7c62-9ce5-4a85-96df-00a6ea763e49-kube-api-access-stlhp\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.047073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.047010    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tmp-dir\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.047164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.047138    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-model-cache\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.047212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.047187    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tls-certs\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.107610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.107569    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.149128    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tmp-dir\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.149224    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-model-cache\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.149266    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tls-certs\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.149359    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-home\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.149407    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-dshm\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.149494    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stlhp\" (UniqueName: \"kubernetes.io/projected/60cb7c62-9ce5-4a85-96df-00a6ea763e49-kube-api-access-stlhp\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.150382    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tmp-dir\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.150701    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-model-cache\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.155678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.151988    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-home\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.162324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.162270    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tls-certs\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.164153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.164126    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-dshm\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.186349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.186319    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stlhp\" (UniqueName: \"kubernetes.io/projected/60cb7c62-9ce5-4a85-96df-00a6ea763e49-kube-api-access-stlhp\") pod \"premium-w0-6af50f-kserve-758c89b79b-zcdt5\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") " pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.310805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.310711    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:23.613783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.613751    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"]
Nov 02 15:07:23.615891 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:07:23.615861    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60cb7c62_9ce5_4a85_96df_00a6ea763e49.slice/crio-eb1f3ff3571920c3a65061b0eb98e6b77dcfee39591d6cd2c7a12e343543ca35 WatchSource:0}: Error finding container eb1f3ff3571920c3a65061b0eb98e6b77dcfee39591d6cd2c7a12e343543ca35: Status 404 returned error can't find the container with id eb1f3ff3571920c3a65061b0eb98e6b77dcfee39591d6cd2c7a12e343543ca35
Nov 02 15:07:23.689986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:23.689950    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:24.278300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:24.278259    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" event={"ID":"60cb7c62-9ce5-4a85-96df-00a6ea763e49","Type":"ContainerStarted","Data":"3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7"}
Nov 02 15:07:24.278814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:24.278313    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" event={"ID":"60cb7c62-9ce5-4a85-96df-00a6ea763e49","Type":"ContainerStarted","Data":"eb1f3ff3571920c3a65061b0eb98e6b77dcfee39591d6cd2c7a12e343543ca35"}
Nov 02 15:07:24.311229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:24.311188    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:24.321053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:24.321004    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podStartSLOduration=2.32098938 podStartE2EDuration="2.32098938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:24.320681334 +0000 UTC m=+638.574634960" watchObservedRunningTime="2026-11-02 15:07:24.32098938 +0000 UTC m=+638.574943005"
Nov 02 15:07:25.283173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:25.283138    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:26.804242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:26.804169    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:26.950945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:26.950846    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:27.930145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:27.930104    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:07:28.491221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:28.491180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:30.284044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:30.284008    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:31.873570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:31.873524    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:33.311452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:33.311417    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:33.324696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:33.324666    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:33.344077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:33.344047    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:07:33.532181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:33.532145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:33.755395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:33.755354    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:35.081136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:35.081100    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:36.587629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:36.587590    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:37.637175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.637145    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"]
Nov 02 15:07:37.640498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.640471    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.660437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.660409    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"]
Nov 02 15:07:37.670830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.670800    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7ecfe1\"/\"premium-w4-7ecfe1-kserve-self-signed-certs\""
Nov 02 15:07:37.714692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.714084    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:37.820686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.817391    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-home\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.820686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.817452    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-model-cache\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.820686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.817516    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b568cb-f879-4551-af12-9aecf54d8f6f-tls-certs\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.820686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.817601    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-tmp-dir\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.820686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.817635    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz98q\" (UniqueName: \"kubernetes.io/projected/d7b568cb-f879-4551-af12-9aecf54d8f6f-kube-api-access-nz98q\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.820686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.817778    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-dshm\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.918529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.918438    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-dshm\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.918529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.918494    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-home\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.918757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.918548    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-model-cache\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.918757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.918608    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b568cb-f879-4551-af12-9aecf54d8f6f-tls-certs\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.918757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.918700    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-tmp-dir\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.918757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.918733    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz98q\" (UniqueName: \"kubernetes.io/projected/d7b568cb-f879-4551-af12-9aecf54d8f6f-kube-api-access-nz98q\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.919133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.919069    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-model-cache\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.919133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.919080    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-home\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.919299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.919232    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-tmp-dir\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.921003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.920973    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-dshm\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.921459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.921439    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b568cb-f879-4551-af12-9aecf54d8f6f-tls-certs\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.942777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.942743    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz98q\" (UniqueName: \"kubernetes.io/projected/d7b568cb-f879-4551-af12-9aecf54d8f6f-kube-api-access-nz98q\") pod \"premium-w4-7ecfe1-kserve-7bc6648fff-nw47s\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") " pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:37.953440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:37.953411    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:38.173022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.172931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:38.259320 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:07:38.259281    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7b568cb_f879_4551_af12_9aecf54d8f6f.slice/crio-22d558815b650de027bf6bf97795c9cc938ab72101c6c137c051c3b6cf01e11e WatchSource:0}: Error finding container 22d558815b650de027bf6bf97795c9cc938ab72101c6c137c051c3b6cf01e11e: Status 404 returned error can't find the container with id 22d558815b650de027bf6bf97795c9cc938ab72101c6c137c051c3b6cf01e11e
Nov 02 15:07:38.259521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.259497    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"]
Nov 02 15:07:38.368404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.367728    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" event={"ID":"d7b568cb-f879-4551-af12-9aecf54d8f6f","Type":"ContainerStarted","Data":"2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14"}
Nov 02 15:07:38.368404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.367785    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" event={"ID":"d7b568cb-f879-4551-af12-9aecf54d8f6f","Type":"ContainerStarted","Data":"22d558815b650de027bf6bf97795c9cc938ab72101c6c137c051c3b6cf01e11e"}
Nov 02 15:07:38.412096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.412041    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podStartSLOduration=1.412026719 podStartE2EDuration="1.412026719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:38.409798309 +0000 UTC m=+652.663751934" watchObservedRunningTime="2026-11-02 15:07:38.412026719 +0000 UTC m=+652.665980344"
Nov 02 15:07:38.727492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.727451    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:38.883756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.883716    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"]
Nov 02 15:07:38.886912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.886889    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:38.902425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.902395    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6af50f-openshift-default-dockercfg-n5jqm\""
Nov 02 15:07:38.911129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.911095    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"]
Nov 02 15:07:38.953907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:38.953871    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:39.032249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032163    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032209    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032231    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2br\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-kube-api-access-hz2br\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032429    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032457    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032498    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032624    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032706    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.032775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.032760    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133347    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133405    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133577    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133711    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133757    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133801    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133840    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2br\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-kube-api-access-hz2br\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.133936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133920    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.134004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.133954    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.134057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.134003    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.134140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.134115    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.134256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.134224    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.134381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.134289    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.134381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.134297    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.136146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.136125    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.136489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.136465    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.157414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.157381    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.157558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.157542    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2br\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-kube-api-access-hz2br\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.200203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.200165    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:39.517638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.517599    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"]
Nov 02 15:07:39.518760 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:07:39.518731    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d403ee_1f73_4b4a_bcb6_85833d42e737.slice/crio-e99e079e4469441ca030f26d170462a9cf3963c5214db7be37fe989cace1478a WatchSource:0}: Error finding container e99e079e4469441ca030f26d170462a9cf3963c5214db7be37fe989cace1478a: Status 404 returned error can't find the container with id e99e079e4469441ca030f26d170462a9cf3963c5214db7be37fe989cace1478a
Nov 02 15:07:39.521257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.521220    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:39.521341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.521286    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:39.521341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.521315    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:39.879982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:39.879940    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:40.398988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:40.398940    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" event={"ID":"d7d403ee-1f73-4b4a-bcb6-85833d42e737","Type":"ContainerStarted","Data":"b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751"}
Nov 02 15:07:40.399192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:40.399012    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" event={"ID":"d7d403ee-1f73-4b4a-bcb6-85833d42e737","Type":"ContainerStarted","Data":"e99e079e4469441ca030f26d170462a9cf3963c5214db7be37fe989cace1478a"}
Nov 02 15:07:40.458512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:40.458456    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podStartSLOduration=2.458441551 podStartE2EDuration="2.458441551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:40.456227657 +0000 UTC m=+654.710181281" watchObservedRunningTime="2026-11-02 15:07:40.458441551 +0000 UTC m=+654.712395175"
Nov 02 15:07:41.202118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:41.202072    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:41.203748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:41.203706    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 15:07:41.203897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:41.203791    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 15:07:41.696948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:41.696894    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:42.201207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:42.201168    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 15:07:42.201413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:42.201234    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 15:07:43.201039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:43.201005    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 15:07:43.201567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:43.201063    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 15:07:43.388335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:43.388302    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:43.595331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:43.595191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:44.202130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:44.202087    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 15:07:44.202635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:44.202160    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 15:07:45.109546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:45.109505    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:45.201104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:45.201066    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 02 15:07:45.201305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:45.201133    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 02 15:07:45.930378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:45.930346    7019 scope.go:122] "RemoveContainer" containerID="d734a12ea770808ab5ed643bb0dd361d01b1fcd9bada3a6e66510c8f8f648911"
Nov 02 15:07:46.323622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:46.323541    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:46.323921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:46.323893    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:46.324749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:46.324729    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:07:46.703324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:46.703287    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:47.954085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:47.954049    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:47.966852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:47.966821    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:48.291776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:48.291678    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:48.483862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:48.483831    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:07:49.888486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:49.888448    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:50.898030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:50.897987    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:07:51.490538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:51.490503    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:52.706215 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:07:52.706176    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:07:53.348961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.348928    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"]
Nov 02 15:07:53.352733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.352717    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.368838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.368813    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-624254-openshift-default-dockercfg-c82tf\""
Nov 02 15:07:53.388623 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.388582    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"]
Nov 02 15:07:53.409060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.409021    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:53.418264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418227    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418290    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418428    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dcb116-9946-41b3-9401-193076a9b410-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418487    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5kv\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-kube-api-access-nw5kv\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418593    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418634    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dcb116-9946-41b3-9401-193076a9b410-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418812    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418858    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.418986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.418922    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520209    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520277    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520312    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520367    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dcb116-9946-41b3-9401-193076a9b410-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520410    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5kv\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-kube-api-access-nw5kv\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520443    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520467    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dcb116-9946-41b3-9401-193076a9b410-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520568    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520615    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520735    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.520908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520832    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.521018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.520986    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.521230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.521209    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.521351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.521317    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dcb116-9946-41b3-9401-193076a9b410-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.523003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.522975    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.523191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.523173    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dcb116-9946-41b3-9401-193076a9b410-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.545188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.545161    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.545435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.545414    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5kv\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-kube-api-access-nw5kv\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.666888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.666852    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:53.756409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.756377    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"]
Nov 02 15:07:53.761358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.760574    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.775432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.775400    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-739bf1-openshift-default-dockercfg-dm45l\""
Nov 02 15:07:53.785014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.784928    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"]
Nov 02 15:07:53.827958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.827918    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.827986    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.828090    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc15e21c-0124-4eff-95e5-234166fb9afc-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.828147    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.828239    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.828282    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.828344    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.828363    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29sc2\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-kube-api-access-29sc2\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.828461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.828449    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929124    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929178    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929208    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929247    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929268    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29sc2\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-kube-api-access-29sc2\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929299    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929345    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929424    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929475    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc15e21c-0124-4eff-95e5-234166fb9afc-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929591    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929685    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.929873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929713    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.930014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.929997    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.930189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.930170    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc15e21c-0124-4eff-95e5-234166fb9afc-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.931794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.931774    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.932120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.932098    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.953140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.953116    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.953277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.953148    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29sc2\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-kube-api-access-29sc2\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:53.993754 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:07:53.993721    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dcb116_9946_41b3_9401_193076a9b410.slice/crio-ef8aba23a241af69ee7c3b17e3cec3f1ac498370141b2110c2dafe94eb2d8d06 WatchSource:0}: Error finding container ef8aba23a241af69ee7c3b17e3cec3f1ac498370141b2110c2dafe94eb2d8d06: Status 404 returned error can't find the container with id ef8aba23a241af69ee7c3b17e3cec3f1ac498370141b2110c2dafe94eb2d8d06
Nov 02 15:07:53.994344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.994315    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"]
Nov 02 15:07:53.997006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.996966    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:53.997108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.997055    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:53.997108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:53.997101    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:54.077559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.077527    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:54.411086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.411056    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"]
Nov 02 15:07:54.411752 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:07:54.411720    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc15e21c_0124_4eff_95e5_234166fb9afc.slice/crio-de27c4e6ca4ca51f831842bc45927f22e3d3371b8d62309b663a4e15ab20e014 WatchSource:0}: Error finding container de27c4e6ca4ca51f831842bc45927f22e3d3371b8d62309b663a4e15ab20e014: Status 404 returned error can't find the container with id de27c4e6ca4ca51f831842bc45927f22e3d3371b8d62309b663a4e15ab20e014
Nov 02 15:07:54.414539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.414501    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:54.414620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.414589    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:54.414692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.414632    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:07:54.514856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.514815    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" event={"ID":"fc15e21c-0124-4eff-95e5-234166fb9afc","Type":"ContainerStarted","Data":"ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e"}
Nov 02 15:07:54.515061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.514889    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" event={"ID":"fc15e21c-0124-4eff-95e5-234166fb9afc","Type":"ContainerStarted","Data":"de27c4e6ca4ca51f831842bc45927f22e3d3371b8d62309b663a4e15ab20e014"}
Nov 02 15:07:54.518205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.518164    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" event={"ID":"a0dcb116-9946-41b3-9401-193076a9b410","Type":"ContainerStarted","Data":"b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de"}
Nov 02 15:07:54.518332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.518233    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" event={"ID":"a0dcb116-9946-41b3-9401-193076a9b410","Type":"ContainerStarted","Data":"ef8aba23a241af69ee7c3b17e3cec3f1ac498370141b2110c2dafe94eb2d8d06"}
Nov 02 15:07:54.566129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.566075    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podStartSLOduration=1.5660599419999999 podStartE2EDuration="1.566059942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:54.564143779 +0000 UTC m=+668.818097405" watchObservedRunningTime="2026-11-02 15:07:54.566059942 +0000 UTC m=+668.820013566"
Nov 02 15:07:54.611757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.611692    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podStartSLOduration=1.611677147 podStartE2EDuration="1.611677147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:54.60895848 +0000 UTC m=+668.862912126" watchObservedRunningTime="2026-11-02 15:07:54.611677147 +0000 UTC m=+668.865630842"
Nov 02 15:07:54.667902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:54.667819    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:07:55.059119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:55.059028    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:55.096269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:55.096192    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:07:55.120671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:55.103890    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:07:55.120671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:55.104003    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 15:07:55.670725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:55.670687    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:07:55.671058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:55.671018    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 15:07:56.081735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:56.081496    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:07:56.081735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:56.081580    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 15:07:56.672671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:56.669068    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:07:56.672671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:56.669134    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 15:07:56.703672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:56.702675    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:57.098736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:57.085039    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:07:57.098736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:57.085108    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 15:07:57.668669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:57.668089    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:07:57.668669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:57.668155    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 15:07:58.095129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:58.095027    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:07:58.095129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:58.095089    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 15:07:58.321363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:58.321312    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:58.661494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:58.661456    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:07:58.669054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:58.669017    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:07:58.669354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:58.669313    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 15:07:59.092184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:59.092067    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:07:59.092460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:59.092413    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 15:07:59.669686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:59.669089    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:07:59.669686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:59.669161    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 15:07:59.888675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:07:59.886619    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:00.084736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:00.084183    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:08:00.084736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:00.084247    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 15:08:00.667789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:00.667743    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 02 15:08:00.667998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:00.667817    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 02 15:08:01.078784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:01.078636    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body=
Nov 02 15:08:01.079254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:01.079182    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused"
Nov 02 15:08:01.520582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:01.520542    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:01.671855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:01.671823    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:08:01.672234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:01.672215    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:08:01.672986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:01.672967    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:08:02.082366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:02.082332    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:08:02.082817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:02.082752    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:08:02.083679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:02.083660    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:08:03.095014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:03.094978    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:03.244849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:03.244800    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:04.795707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:04.795674    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:06.493160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:06.493123    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:08.199911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:08.199874    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:10.108709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:10.108674    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:11.856069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:11.856032    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:13.446934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:13.443520    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:15.007248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:15.007211    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:16.586704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:16.586665    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:16.876557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:16.876518    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:08:18.372878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:18.372837    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:19.970326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:19.970287    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:21.585240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:21.585180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:22.706091 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:08:22.706045    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:08:23.217590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.217550    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:23.735986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.735944    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"]
Nov 02 15:08:23.741237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.741218    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.758043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.758018    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5dd8e9-openshift-default-dockercfg-fthlq\""
Nov 02 15:08:23.773840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.773797    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"]
Nov 02 15:08:23.853660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.853615    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.853852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.853783    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.853917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.853885    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplcd\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-kube-api-access-xplcd\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.853917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.853912    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.854018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.853928    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.854072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.854012    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.854072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.854052    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.854180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.854071    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab090f8-be90-4abf-b504-35919d67bb0a-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.854238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.854176    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab090f8-be90-4abf-b504-35919d67bb0a-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955385    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplcd\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-kube-api-access-xplcd\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955427    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955444    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955695    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955756    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955786    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab090f8-be90-4abf-b504-35919d67bb0a-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955819    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955840    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab090f8-be90-4abf-b504-35919d67bb0a-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.955865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.955851    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.956065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.956024    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.956065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.956056    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.956177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.956162    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.956561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.956366    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.956561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.956532    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab090f8-be90-4abf-b504-35919d67bb0a-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.958003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.957984    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.958322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.958305    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab090f8-be90-4abf-b504-35919d67bb0a-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.984941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.984909    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:23.985101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:23.985009    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplcd\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-kube-api-access-xplcd\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:24.055382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.055296    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:24.376996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.376953    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"]
Nov 02 15:08:24.378475 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:08:24.378444    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab090f8_be90_4abf_b504_35919d67bb0a.slice/crio-cb8315a39ed0e84532f12b1e8463cbd1a1bbcf2279deef4873221d00df5cb68c WatchSource:0}: Error finding container cb8315a39ed0e84532f12b1e8463cbd1a1bbcf2279deef4873221d00df5cb68c: Status 404 returned error can't find the container with id cb8315a39ed0e84532f12b1e8463cbd1a1bbcf2279deef4873221d00df5cb68c
Nov 02 15:08:24.381535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.381489    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:24.381685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.381587    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:24.381685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.381636    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:08:24.740111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.740012    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" event={"ID":"dab090f8-be90-4abf-b504-35919d67bb0a","Type":"ContainerStarted","Data":"966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8"}
Nov 02 15:08:24.740111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.740094    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" event={"ID":"dab090f8-be90-4abf-b504-35919d67bb0a","Type":"ContainerStarted","Data":"cb8315a39ed0e84532f12b1e8463cbd1a1bbcf2279deef4873221d00df5cb68c"}
Nov 02 15:08:24.799941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:24.799886    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podStartSLOduration=1.7998702309999999 podStartE2EDuration="1.799870231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:24.797683011 +0000 UTC m=+699.051636635" watchObservedRunningTime="2026-11-02 15:08:24.799870231 +0000 UTC m=+699.053823922"
Nov 02 15:08:25.055567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:25.055468    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:25.056962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:25.056938    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:08:25.057041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:25.056990    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 15:08:25.099672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:25.092496    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:26.056056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:26.056014    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:08:26.056462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:26.056087    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 15:08:26.721776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:26.721727    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:27.056004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:27.055916    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:08:27.056004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:27.055991    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 15:08:28.056130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:28.056094    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:08:28.056321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:28.056156    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 15:08:28.506515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:28.506425    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:29.056525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:29.056489    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:08:29.056958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:29.056548    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 15:08:30.055670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:30.055625    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 15:08:30.055882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:30.055701    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 15:08:30.115868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:30.115828    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:31.062474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:31.062439    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:31.062883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:31.062865    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:31.063880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:31.063850    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:08:31.784272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:31.784235    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:33.504492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:33.504457    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:35.194250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:35.194207    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:36.893628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:36.893581    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:38.733707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:38.733665    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:38.878139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:38.878102    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:40.778332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:40.778294    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:41.428710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:41.419596    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:43.067504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:43.067468    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:43.708452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:43.708411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:44.934604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:44.934568    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:46.671013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:46.670970    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:47.472909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:47.472868    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:08:48.436699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:48.436638    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:50.532123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:50.532085    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:52.300475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:52.300434    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:52.706157 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:08:52.706124    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:08:53.885044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:53.885007    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:55.512500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:55.512453    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:57.392586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:57.392497    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:58.805082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:58.805048    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:59.089152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:59.089054    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:08:59.595794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:08:59.595752    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:00.901591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:00.901545    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:02.595752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:02.595703    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:02.833481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:02.833450    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:04.232738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:04.232699    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:05.897494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:05.897450    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:06.893632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.893571    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"]
Nov 02 15:09:06.898560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.898534    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.902715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.902637    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.902988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.902967    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.903121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.903104    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59thr\" (UniqueName: \"kubernetes.io/projected/d1affeef-d790-45c6-b74c-9053b4f08f57-kube-api-access-59thr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.903347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.903299    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.903545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.903519    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1affeef-d790-45c6-b74c-9053b4f08f57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.903765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.903745    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.903951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.903933    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.904087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.904071    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.904253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.904236    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:06.930731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:06.930684    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"]
Nov 02 15:09:07.005184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005151    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59thr\" (UniqueName: \"kubernetes.io/projected/d1affeef-d790-45c6-b74c-9053b4f08f57-kube-api-access-59thr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005200    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005240    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1affeef-d790-45c6-b74c-9053b4f08f57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005284    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005353    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005387    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005429    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005470    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005529    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.005862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.005803    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.006277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.006080    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.006277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.006173    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.006277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.006180    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1affeef-d790-45c6-b74c-9053b4f08f57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.006277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.006235    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.008126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.008099    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.008244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.008151    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.029319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.029295    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1affeef-d790-45c6-b74c-9053b4f08f57-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.029444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.029361    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59thr\" (UniqueName: \"kubernetes.io/projected/d1affeef-d790-45c6-b74c-9053b4f08f57-kube-api-access-59thr\") pod \"maas-default-gateway-openshift-default-fcb75c45f-xqbf4\" (UID: \"d1affeef-d790-45c6-b74c-9053b4f08f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.218332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.218242    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:07.536362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.536332    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"]
Nov 02 15:09:07.536567 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:07.536546    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1affeef_d790_45c6_b74c_9053b4f08f57.slice/crio-703ff92f5b1f5d2a5741ed64fe744ec06f226d2e910e6f06516903ecca8110f4 WatchSource:0}: Error finding container 703ff92f5b1f5d2a5741ed64fe744ec06f226d2e910e6f06516903ecca8110f4: Status 404 returned error can't find the container with id 703ff92f5b1f5d2a5741ed64fe744ec06f226d2e910e6f06516903ecca8110f4
Nov 02 15:09:07.538885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.538851    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:07.538998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.538926    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:07.538998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.538971    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:07.753626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:07.753595    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:08.120696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.120639    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" event={"ID":"d1affeef-d790-45c6-b74c-9053b4f08f57","Type":"ContainerStarted","Data":"f7ce623b442107eaa049d2450483f8b0a8d483df71d0d521f9185b5ada1a7dd3"}
Nov 02 15:09:08.121202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.121185    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" event={"ID":"d1affeef-d790-45c6-b74c-9053b4f08f57","Type":"ContainerStarted","Data":"703ff92f5b1f5d2a5741ed64fe744ec06f226d2e910e6f06516903ecca8110f4"}
Nov 02 15:09:08.207825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.207756    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podStartSLOduration=2.207737946 podStartE2EDuration="2.207737946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:08.193134587 +0000 UTC m=+742.447088223" watchObservedRunningTime="2026-11-02 15:09:08.207737946 +0000 UTC m=+742.461691570"
Nov 02 15:09:08.217478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.217440    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:09:08.218466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.218441    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:08.220140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.220110    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:08.220246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.220168    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:08.905382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.905343    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"]
Nov 02 15:09:08.912261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.912229    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:08.931701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:08.931478    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"]
Nov 02 15:09:09.041115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.040704    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frgqk\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-kube-api-access-frgqk\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.041115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.040770    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893f1c33-0434-4804-ab42-1d27c60bb1af-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.041115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.040875    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.041115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.040915    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.041115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.041004    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.041115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.041067    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.044979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.041155    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.044979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.041241    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.044979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.041294    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142110    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142154    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142186    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142306    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142368    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142433    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frgqk\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-kube-api-access-frgqk\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142465    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893f1c33-0434-4804-ab42-1d27c60bb1af-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142579    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142609    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.142672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.142631    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.143109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.143040    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.143192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.143160    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.143509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.143461    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.144404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.143673    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893f1c33-0434-4804-ab42-1d27c60bb1af-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.145119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.145096    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.145812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.145788    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.202841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.184369    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frgqk\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-kube-api-access-frgqk\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.214682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.214419    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.219201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.219164    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:09.219371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.219227    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:09.229184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.229149    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:09.724685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.724229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:09.847405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.846125    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"]
Nov 02 15:09:09.847405 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:09.846493    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893f1c33_0434_4804_ab42_1d27c60bb1af.slice/crio-99f21746a467a912fc716ad5e3a8bd9e2bc535ca709fc758013dd1a85fc38f41 WatchSource:0}: Error finding container 99f21746a467a912fc716ad5e3a8bd9e2bc535ca709fc758013dd1a85fc38f41: Status 404 returned error can't find the container with id 99f21746a467a912fc716ad5e3a8bd9e2bc535ca709fc758013dd1a85fc38f41
Nov 02 15:09:09.851692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.850146    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.851692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.850233    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:09.851692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:09.850279    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:10.236680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:10.233511    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:10.236680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:10.233576    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:10.246102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:10.246065    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" event={"ID":"893f1c33-0434-4804-ab42-1d27c60bb1af","Type":"ContainerStarted","Data":"99f21746a467a912fc716ad5e3a8bd9e2bc535ca709fc758013dd1a85fc38f41"}
Nov 02 15:09:11.221105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:11.221064    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:11.221421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:11.221390    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:11.252532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:11.252501    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" event={"ID":"893f1c33-0434-4804-ab42-1d27c60bb1af","Type":"ContainerStarted","Data":"643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196"}
Nov 02 15:09:11.333677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:11.331998    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podStartSLOduration=3.331975865 podStartE2EDuration="3.331975865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:11.324032902 +0000 UTC m=+745.577986550" watchObservedRunningTime="2026-11-02 15:09:11.331975865 +0000 UTC m=+745.585929490"
Nov 02 15:09:11.616372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:11.616292    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:12.230073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:12.223094    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.230073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:12.223161    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:12.232006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:12.231972    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:12.232781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:12.232753    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:12.232880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:12.232808    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 02 15:09:13.223679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:13.223073    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.223679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:13.223141    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:13.231053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:13.231017    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:13.231195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:13.231083    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 02 15:09:13.582732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:13.580274    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:14.219695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:14.219635    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.219917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:14.219713    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:14.230065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:14.230033    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:14.230429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:14.230084    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 02 15:09:15.219351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:15.219314    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.219550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:15.219376    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Nov 02 15:09:15.230241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:15.230206    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:15.230622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:15.230265    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 02 15:09:15.591387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:15.591300    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:16.229958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:16.229921    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Nov 02 15:09:16.230148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:16.229985    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Nov 02 15:09:16.776880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:16.776847    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:16.778211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:16.778073    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:16.779034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:16.779016    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4"
Nov 02 15:09:17.415628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:17.414892    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:17.416533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:17.416259    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:17.417726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:17.417707    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:09:17.606144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:17.606109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:18.060387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.060352    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"]
Nov 02 15:09:18.065023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.064999    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.100148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.100116    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"]
Nov 02 15:09:18.100721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.100700    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"kube-root-ca.crt\""
Nov 02 15:09:18.107164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.107140    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"distinct-w6-5113c2-kserve-self-signed-certs\""
Nov 02 15:09:18.107164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.107201    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"openshift-service-ca.crt\""
Nov 02 15:09:18.107446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.107301    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"default-dockercfg-dn9b4\""
Nov 02 15:09:18.149359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.149321    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-dshm\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.149569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.149380    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gf9\" (UniqueName: \"kubernetes.io/projected/7f649146-86f7-45f2-b873-afc104a64c1c-kube-api-access-t4gf9\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.149569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.149516    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f649146-86f7-45f2-b873-afc104a64c1c-tls-certs\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.149739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.149568    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-model-cache\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.149739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.149710    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-home\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.149846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.149739    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-tmp-dir\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.158464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.158429    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:18.252672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.251239    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-model-cache\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.252672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.251453    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-home\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.252672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.251504    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-tmp-dir\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.252672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.251601    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-dshm\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.252672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.251665    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gf9\" (UniqueName: \"kubernetes.io/projected/7f649146-86f7-45f2-b873-afc104a64c1c-kube-api-access-t4gf9\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.252672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.251724    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f649146-86f7-45f2-b873-afc104a64c1c-tls-certs\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.253127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.252714    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-tmp-dir\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.253127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.253002    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-model-cache\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.253243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.253230    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-home\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.269357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.257735    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-dshm\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.301585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.299344    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f649146-86f7-45f2-b873-afc104a64c1c-tls-certs\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.301585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.299355    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gf9\" (UniqueName: \"kubernetes.io/projected/7f649146-86f7-45f2-b873-afc104a64c1c-kube-api-access-t4gf9\") pod \"distinct-w6-5113c2-kserve-84b6976588-29k58\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.379923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.379887    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:18.691072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:18.691043    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"]
Nov 02 15:09:18.691996 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:18.691967    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f649146_86f7_45f2_b873_afc104a64c1c.slice/crio-11104c2236eee683187fb68a176af9099a51f8fdcbec114653c1d4e8038e30dc WatchSource:0}: Error finding container 11104c2236eee683187fb68a176af9099a51f8fdcbec114653c1d4e8038e30dc: Status 404 returned error can't find the container with id 11104c2236eee683187fb68a176af9099a51f8fdcbec114653c1d4e8038e30dc
Nov 02 15:09:19.345470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:19.345430    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" event={"ID":"7f649146-86f7-45f2-b873-afc104a64c1c","Type":"ContainerStarted","Data":"65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae"}
Nov 02 15:09:19.345470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:19.345477    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" event={"ID":"7f649146-86f7-45f2-b873-afc104a64c1c","Type":"ContainerStarted","Data":"11104c2236eee683187fb68a176af9099a51f8fdcbec114653c1d4e8038e30dc"}
Nov 02 15:09:19.378333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:19.378301    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:19.380017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:19.379986    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:19.395240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:19.395179    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podStartSLOduration=1.395159997 podStartE2EDuration="1.395159997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:19.394761123 +0000 UTC m=+753.648714747" watchObservedRunningTime="2026-11-02 15:09:19.395159997 +0000 UTC m=+753.649113621"
Nov 02 15:09:21.118214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:21.118173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:22.707097 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:09:22.707050    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:09:22.832347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:22.832314    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:23.320010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.319978    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:23.637313 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.637279    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"]
Nov 02 15:09:23.641996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.641969    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.666278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.665920    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"]
Nov 02 15:09:23.726060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726014    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726082    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726182    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726264    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5kk\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-kube-api-access-ft5kk\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726366    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6348ed9-1285-4509-8a25-516358d3d83e-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726387    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726436    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726528    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6348ed9-1285-4509-8a25-516358d3d83e-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.726872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.726596    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828422    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5kk\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-kube-api-access-ft5kk\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828490    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6348ed9-1285-4509-8a25-516358d3d83e-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828521    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828596    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828642    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6348ed9-1285-4509-8a25-516358d3d83e-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828738    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828801    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828854    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.828915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.828900    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.829201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.829171    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.829275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.829195    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.829464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.829438    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.829553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.829512    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.829602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.829557    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6348ed9-1285-4509-8a25-516358d3d83e-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.831098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.831069    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.831224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.831208    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6348ed9-1285-4509-8a25-516358d3d83e-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.851907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.851883    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.852042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.852019    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5kk\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-kube-api-access-ft5kk\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:23.959477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:23.959391    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:24.284175 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:24.284140    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6348ed9_1285_4509_8a25_516358d3d83e.slice/crio-c48384197c07fa3dd611ac12672e4755ba4eb21bd7ad8f8d678a82a6a0ff6e68 WatchSource:0}: Error finding container c48384197c07fa3dd611ac12672e4755ba4eb21bd7ad8f8d678a82a6a0ff6e68: Status 404 returned error can't find the container with id c48384197c07fa3dd611ac12672e4755ba4eb21bd7ad8f8d678a82a6a0ff6e68
Nov 02 15:09:24.284325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.284262    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"]
Nov 02 15:09:24.289601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.287961    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:24.289601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.288053    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:24.289601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.288093    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:24.390672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.390291    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" event={"ID":"e6348ed9-1285-4509-8a25-516358d3d83e","Type":"ContainerStarted","Data":"ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d"}
Nov 02 15:09:24.390672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.390354    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" event={"ID":"e6348ed9-1285-4509-8a25-516358d3d83e","Type":"ContainerStarted","Data":"c48384197c07fa3dd611ac12672e4755ba4eb21bd7ad8f8d678a82a6a0ff6e68"}
Nov 02 15:09:24.435442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.435242    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podStartSLOduration=1.435222027 podStartE2EDuration="1.435222027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:24.434785773 +0000 UTC m=+758.688739398" watchObservedRunningTime="2026-11-02 15:09:24.435222027 +0000 UTC m=+758.689175652"
Nov 02 15:09:24.692126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.692043    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:24.960511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:24.960433    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:25.962279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:25.962245    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:25.962779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:25.962305    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 15:09:26.524159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:26.524122    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:26.959957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:26.959912    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:26.960173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:26.959976    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 15:09:27.354226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.354133    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"]
Nov 02 15:09:27.359898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.359868    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.404669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.398640    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"]
Nov 02 15:09:27.404669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.402140    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"openshift-service-ca.crt\""
Nov 02 15:09:27.404669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.402253    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"distinct-w1-5dd8e9-kserve-self-signed-certs\""
Nov 02 15:09:27.404669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.402322    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"kube-root-ca.crt\""
Nov 02 15:09:27.404669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.402404    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5dd8e9\"/\"default-dockercfg-ccmk6\""
Nov 02 15:09:27.454697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.454659    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:27.496673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.496374    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af87adc8-9bc6-4766-9850-45cb1e77469b-tls-certs\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.496673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.496445    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-model-cache\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.496673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.496504    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5lhp\" (UniqueName: \"kubernetes.io/projected/af87adc8-9bc6-4766-9850-45cb1e77469b-kube-api-access-r5lhp\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.496673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.496544    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-home\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.496673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.496640    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-tmp-dir\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.497154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.496738    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-dshm\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.601427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.601385    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af87adc8-9bc6-4766-9850-45cb1e77469b-tls-certs\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.602208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.602181    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-model-cache\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.602416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.602397    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5lhp\" (UniqueName: \"kubernetes.io/projected/af87adc8-9bc6-4766-9850-45cb1e77469b-kube-api-access-r5lhp\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.602537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.602524    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-home\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.602756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.602739    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-tmp-dir\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.602934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.602918    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-dshm\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.618748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.604265    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-model-cache\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.618748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.604511    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-home\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.618748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.604757    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-tmp-dir\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.619389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.619361    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-dshm\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.628301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.628266    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af87adc8-9bc6-4766-9850-45cb1e77469b-tls-certs\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.637321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.637290    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5lhp\" (UniqueName: \"kubernetes.io/projected/af87adc8-9bc6-4766-9850-45cb1e77469b-kube-api-access-r5lhp\") pod \"distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") " pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.701933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.701896    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:27.960782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.960701    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:27.960933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:27.960764    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 15:09:28.058252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.058224    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"]
Nov 02 15:09:28.058934 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:28.058909    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf87adc8_9bc6_4766_9850_45cb1e77469b.slice/crio-feb9acb9907cbd3c58b10a5a9c44bffba6361d90dc542af1e670759ef6e8f83c WatchSource:0}: Error finding container feb9acb9907cbd3c58b10a5a9c44bffba6361d90dc542af1e670759ef6e8f83c: Status 404 returned error can't find the container with id feb9acb9907cbd3c58b10a5a9c44bffba6361d90dc542af1e670759ef6e8f83c
Nov 02 15:09:28.388674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.384640    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:28.418111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.417841    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:28.427377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.427342    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" event={"ID":"af87adc8-9bc6-4766-9850-45cb1e77469b","Type":"ContainerStarted","Data":"7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de"}
Nov 02 15:09:28.427534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.427385    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" event={"ID":"af87adc8-9bc6-4766-9850-45cb1e77469b","Type":"ContainerStarted","Data":"feb9acb9907cbd3c58b10a5a9c44bffba6361d90dc542af1e670759ef6e8f83c"}
Nov 02 15:09:28.439800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.439773    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:09:28.470266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.470233    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:28.584723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.584393    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podStartSLOduration=1.58437092 podStartE2EDuration="1.58437092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:28.551105829 +0000 UTC m=+762.805059453" watchObservedRunningTime="2026-11-02 15:09:28.58437092 +0000 UTC m=+762.838324545"
Nov 02 15:09:28.711775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.711743    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:28.961621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.961579    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:28.961906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:28.961878    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 15:09:29.959987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:29.959951    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:29.960396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:29.960010    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 15:09:30.436182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:30.436153    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:30.960259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:30.960216    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:09:30.960766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:30.960279    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 15:09:32.155580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:32.155521    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:32.156313 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:32.155869    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:32.157002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:32.156981    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:09:32.179098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:32.179064    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:34.076993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:34.076961    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:35.690755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:35.690711    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:37.278148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:37.278064    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:37.703217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:37.703180    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:37.720745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:37.720709    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:38.569533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:38.569502    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:09:38.576232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:38.576199    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:09:38.826199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:38.826107    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:40.674518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:40.674471    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:41.185592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:41.185553    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:42.387639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:42.387600    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:43.826880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:43.826815    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:43.991467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:43.991431    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:44.693858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.693825    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"]
Nov 02 15:09:44.700247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.700219    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.715593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.715567    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"]
Nov 02 15:09:44.725198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.725165    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6af50f\"/\"distinct-2-w0-6af50f-kserve-self-signed-certs\""
Nov 02 15:09:44.777554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.777515    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:44.892675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.891641    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a6d0fb-4834-47b4-928f-7baf205291fc-tls-certs\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.892675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.891797    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-model-cache\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.892675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.891822    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-tmp-dir\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.892675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.891852    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr4g8\" (UniqueName: \"kubernetes.io/projected/f9a6d0fb-4834-47b4-928f-7baf205291fc-kube-api-access-sr4g8\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.892675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.891917    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-home\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.892675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.891974    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-dshm\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.993448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.993355    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-model-cache\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.994096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.994077    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-tmp-dir\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.994546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.994531    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr4g8\" (UniqueName: \"kubernetes.io/projected/f9a6d0fb-4834-47b4-928f-7baf205291fc-kube-api-access-sr4g8\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.995074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.994435    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-tmp-dir\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.995163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.994031    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-model-cache\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.995296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.995276    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-home\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.995711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.995696    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-dshm\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.996278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.996262    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a6d0fb-4834-47b4-928f-7baf205291fc-tls-certs\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.996540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.995609    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-home\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.998383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.998327    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-dshm\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:44.999692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:44.999640    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a6d0fb-4834-47b4-928f-7baf205291fc-tls-certs\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:45.019627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:45.019595    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr4g8\" (UniqueName: \"kubernetes.io/projected/f9a6d0fb-4834-47b4-928f-7baf205291fc-kube-api-access-sr4g8\") pod \"distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") " pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:45.315692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:45.315585    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:45.623112 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:45.623084    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9a6d0fb_4834_47b4_928f_7baf205291fc.slice/crio-db357c7b854d85ba281b0ffbb34088194d46a46be4d6f37e26da741fa772f22e WatchSource:0}: Error finding container db357c7b854d85ba281b0ffbb34088194d46a46be4d6f37e26da741fa772f22e: Status 404 returned error can't find the container with id db357c7b854d85ba281b0ffbb34088194d46a46be4d6f37e26da741fa772f22e
Nov 02 15:09:45.624365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:45.624339    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"]
Nov 02 15:09:45.759084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:45.759049    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:46.622940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:46.622902    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" event={"ID":"f9a6d0fb-4834-47b4-928f-7baf205291fc","Type":"ContainerStarted","Data":"d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f"}
Nov 02 15:09:46.623355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:46.622951    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" event={"ID":"f9a6d0fb-4834-47b4-928f-7baf205291fc","Type":"ContainerStarted","Data":"db357c7b854d85ba281b0ffbb34088194d46a46be4d6f37e26da741fa772f22e"}
Nov 02 15:09:46.666248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:46.666195    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podStartSLOduration=2.666178692 podStartE2EDuration="2.666178692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:46.664155698 +0000 UTC m=+780.918109323" watchObservedRunningTime="2026-11-02 15:09:46.666178692 +0000 UTC m=+780.920132315"
Nov 02 15:09:47.316094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:47.316062    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:47.332062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:47.332026    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:48.986462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:48.986424    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:50.591301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:50.591255    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:51.228392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:51.228352    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:52.179359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:52.179327    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:52.710541 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:09:52.710502    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:09:54.049185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.049146    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"]
Nov 02 15:09:54.057575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.057543    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.069148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.066848    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"]
Nov 02 15:09:54.130606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.130569    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.131232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.131202    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.131494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.131475    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a68852-928e-47c0-8213-dbf2019f33b9-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.131709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.131688    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a68852-928e-47c0-8213-dbf2019f33b9-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.131914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.131895    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2fs\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-kube-api-access-rx2fs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.132090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.132074    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.132241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.132226    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.132557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.132499    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.132759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.132741    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.152798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.152763    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"]
Nov 02 15:09:54.157024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.157001    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.181789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.181758    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"]
Nov 02 15:09:54.207355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.207142    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:54.239682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.239600    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.239963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.239936    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.240311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240273    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240338    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240370    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc91941e-07ef-4265-8a68-78be32735b89-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240422    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc91941e-07ef-4265-8a68-78be32735b89-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240463    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.240526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240487    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.240709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240523    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240678    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240711    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.240822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240736    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.240822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240779    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a68852-928e-47c0-8213-dbf2019f33b9-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.240971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240827    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a68852-928e-47c0-8213-dbf2019f33b9-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.240971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240856    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240883    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9knvs\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-kube-api-access-9knvs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240927    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.240971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.240960    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2fs\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-kube-api-access-rx2fs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.241943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.241883    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-credential-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.244851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.242427    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-certs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.245014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.243599    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a68852-928e-47c0-8213-dbf2019f33b9-istiod-ca-cert\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.245145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.243882    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-socket\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.245307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.244456    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-data\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.245743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.245721    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a68852-928e-47c0-8213-dbf2019f33b9-istio-podinfo\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.246457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.246432    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-envoy\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.266344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.266310    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2fs\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-kube-api-access-rx2fs\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.269950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.269914    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-istio-token\") pod \"e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") " pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.342040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.341949    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.341999    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342022    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc91941e-07ef-4265-8a68-78be32735b89-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342068    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc91941e-07ef-4265-8a68-78be32735b89-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342115    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342218    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342274    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342298    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9knvs\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-kube-api-access-9knvs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342336    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342406    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342597    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342671    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342907    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc91941e-07ef-4265-8a68-78be32735b89-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.342930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.342922    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.345548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.345522    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.345777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.345756    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc91941e-07ef-4265-8a68-78be32735b89-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.368018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.367991    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.368154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.368104    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9knvs\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-kube-api-access-9knvs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.402627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.402584    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:54.505700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.498339    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:54.745174 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:54.745145    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a68852_928e_47c0_8213_dbf2019f33b9.slice/crio-4d73eab5d98260c7270340e41e7bc82ea0a6ef2288095c67e461137f0806bfa7 WatchSource:0}: Error finding container 4d73eab5d98260c7270340e41e7bc82ea0a6ef2288095c67e461137f0806bfa7: Status 404 returned error can't find the container with id 4d73eab5d98260c7270340e41e7bc82ea0a6ef2288095c67e461137f0806bfa7
Nov 02 15:09:54.745390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.745367    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"]
Nov 02 15:09:54.747562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.747526    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.747633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.747594    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.747633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.747625    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.856530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.856497    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"]
Nov 02 15:09:54.857540 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:09:54.857511    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc91941e_07ef_4265_8a68_78be32735b89.slice/crio-f298177d1970d56510d6fb4da441246ed7a8b739dc6078d59dc28083a3244adb WatchSource:0}: Error finding container f298177d1970d56510d6fb4da441246ed7a8b739dc6078d59dc28083a3244adb: Status 404 returned error can't find the container with id f298177d1970d56510d6fb4da441246ed7a8b739dc6078d59dc28083a3244adb
Nov 02 15:09:54.861099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.861050    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.861215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.861141    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:54.861215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:54.861182    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:09:55.317673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.316794    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:55.336261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.336224    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:55.720194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.720157    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" event={"ID":"43a68852-928e-47c0-8213-dbf2019f33b9","Type":"ContainerStarted","Data":"2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5"}
Nov 02 15:09:55.720479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.720457    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" event={"ID":"43a68852-928e-47c0-8213-dbf2019f33b9","Type":"ContainerStarted","Data":"4d73eab5d98260c7270340e41e7bc82ea0a6ef2288095c67e461137f0806bfa7"}
Nov 02 15:09:55.729737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.729045    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" event={"ID":"fc91941e-07ef-4265-8a68-78be32735b89","Type":"ContainerStarted","Data":"6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad"}
Nov 02 15:09:55.729737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.729101    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" event={"ID":"fc91941e-07ef-4265-8a68-78be32735b89","Type":"ContainerStarted","Data":"f298177d1970d56510d6fb4da441246ed7a8b739dc6078d59dc28083a3244adb"}
Nov 02 15:09:55.741426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.741392    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:09:55.775705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:55.775625    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podStartSLOduration=1.7756040629999998 podStartE2EDuration="1.775604063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:55.773414231 +0000 UTC m=+790.027367853" watchObservedRunningTime="2026-11-02 15:09:55.775604063 +0000 UTC m=+790.029557669"
Nov 02 15:09:56.107734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:56.101363    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:56.406674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:56.405305    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:09:56.406674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:56.405482    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:09:56.406674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:56.405534    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 15:09:56.508442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:56.508405    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:09:56.508849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:56.508822    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:09:56.509042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:56.509015    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 15:09:57.408674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:57.407112    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:09:57.408674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:57.407180    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 15:09:57.499672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:57.499218    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:09:57.499672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:57.499295    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 15:09:57.946004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:57.945969    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:58.417672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:58.417107    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:09:58.417672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:58.417175    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 15:09:58.501674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:58.501022    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:09:58.501674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:58.501100    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 15:09:58.941127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:58.941082    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:09:59.405094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:59.405057    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:09:59.405363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:59.405335    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 15:09:59.502674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:59.500411    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:09:59.502674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:59.500487    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 15:09:59.739472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:09:59.739391    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:00.411672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:00.411082    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:00.411672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:00.411161    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 15:10:00.511671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:00.510762    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:10:00.511671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:00.510855    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 15:10:01.403413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:01.403372    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 02 15:10:01.403740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:01.403707    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 02 15:10:01.496915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:01.496874    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:01.499906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:01.499873    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 15:10:01.500079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:01.499933    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 15:10:02.408340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:02.408302    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:10:02.408882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:02.408867    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:10:02.409520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:02.409499    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:10:02.487903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:02.487843    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podStartSLOduration=8.487828539 podStartE2EDuration="8.487828539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:09:55.874773398 +0000 UTC m=+790.128727023" watchObservedRunningTime="2026-11-02 15:10:02.487828539 +0000 UTC m=+796.741782163"
Nov 02 15:10:02.503775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:02.503741    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:10:02.504205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:02.504181    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:10:02.504992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:02.504969    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:10:03.222768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:03.222734    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:05.029590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:05.029553    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:06.539533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:06.539494    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:08.176609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.176571    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:08.500318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.500224    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:08.664848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.663744    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"]
Nov 02 15:10:08.670007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.669976    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.684707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.683550    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"]
Nov 02 15:10:08.693126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.693097    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-7ecfe1-openshift-default-dockercfg-gg2h7\""
Nov 02 15:10:08.766334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766224    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d61337-0a44-4b54-8259-45b7c4467893-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.766334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766294    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.766537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766443    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.766537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766493    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.766642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766613    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjgvj\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-kube-api-access-xjgvj\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.766765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766746    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.766934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766915    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.766972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766962    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.767018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.766990    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d61337-0a44-4b54-8259-45b7c4467893-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.868737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.868697    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.868737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.868745    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.868763    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d61337-0a44-4b54-8259-45b7c4467893-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.868796    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d61337-0a44-4b54-8259-45b7c4467893-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.868856    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.869057    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.869106    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.869160    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjgvj\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-kube-api-access-xjgvj\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.869239    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.869688    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d61337-0a44-4b54-8259-45b7c4467893-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.869933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.869908    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.870023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.869930    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.870223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.870202    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.870358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.870339    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.871945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.871905    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.871945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.871906    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d61337-0a44-4b54-8259-45b7c4467893-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.895377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.895344    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.895536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.895464    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjgvj\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-kube-api-access-xjgvj\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:08.990091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:08.990051    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:09.315831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.315801    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"]
Nov 02 15:10:09.316662 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:10:09.316620    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d61337_0a44_4b54_8259_45b7c4467893.slice/crio-5f8879ce4f7304fa8dbb087b56a261378a8ac3a51037c552242b5d0232d81325 WatchSource:0}: Error finding container 5f8879ce4f7304fa8dbb087b56a261378a8ac3a51037c552242b5d0232d81325: Status 404 returned error can't find the container with id 5f8879ce4f7304fa8dbb087b56a261378a8ac3a51037c552242b5d0232d81325
Nov 02 15:10:09.319202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.319167    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:09.319315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.319235    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:09.319315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.319264    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:09.417941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.417909    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"]
Nov 02 15:10:09.422203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.422174    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.451447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.451416    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"]
Nov 02 15:10:09.486570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486501    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.486885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486606    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.486885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486662    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.486885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486686    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7qz\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-kube-api-access-bx7qz\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.486885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486745    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.486885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486791    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.486885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486851    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.486885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.486877    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.487573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.487068    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398f9eb8-3c21-4f44-8172-bd99b4618c40-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.588636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588554    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.588636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588586    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.588885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588728    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398f9eb8-3c21-4f44-8172-bd99b4618c40-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.588885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588768    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.588885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588834    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.588885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588865    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588892    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7qz\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-kube-api-access-bx7qz\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588943    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.588988    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.589165    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.589333    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.589478    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398f9eb8-3c21-4f44-8172-bd99b4618c40-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.589506    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.589668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.589563    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.591536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.591510    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.591820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.591802    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.614312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.614279    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.614516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.614495    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7qz\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-kube-api-access-bx7qz\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.741609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.741367    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:09.897063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.897016    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" event={"ID":"45d61337-0a44-4b54-8259-45b7c4467893","Type":"ContainerStarted","Data":"6ebdc2e00691930b42a59a9649ae2fb246d661804cd933f0ba0e89139572d5f1"}
Nov 02 15:10:09.902673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.897514    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" event={"ID":"45d61337-0a44-4b54-8259-45b7c4467893","Type":"ContainerStarted","Data":"5f8879ce4f7304fa8dbb087b56a261378a8ac3a51037c552242b5d0232d81325"}
Nov 02 15:10:09.932816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.929941    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:10.000673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:09.998754    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:10.096292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:10.096196    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podStartSLOduration=2.096180071 podStartE2EDuration="2.096180071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:09.971433252 +0000 UTC m=+804.225386877" watchObservedRunningTime="2026-11-02 15:10:10.096180071 +0000 UTC m=+804.350133733"
Nov 02 15:10:10.098938 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:10:10.098907    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398f9eb8_3c21_4f44_8172_bd99b4618c40.slice/crio-8d498764486bdfa592718beae20ef70fb4b7941f3dcf1a414e1d75ade3702656 WatchSource:0}: Error finding container 8d498764486bdfa592718beae20ef70fb4b7941f3dcf1a414e1d75ade3702656: Status 404 returned error can't find the container with id 8d498764486bdfa592718beae20ef70fb4b7941f3dcf1a414e1d75ade3702656
Nov 02 15:10:10.099385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:10.099363    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"]
Nov 02 15:10:10.102214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:10.102179    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:10.102321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:10.102247    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:10.102321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:10.102275    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:10.917674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:10.904989    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" event={"ID":"398f9eb8-3c21-4f44-8172-bd99b4618c40","Type":"ContainerStarted","Data":"5226135c192c91ac56cffdbab53b0f1c7adf80623e4d3d5bd540530ad0044a2c"}
Nov 02 15:10:10.917674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:10.905053    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" event={"ID":"398f9eb8-3c21-4f44-8172-bd99b4618c40","Type":"ContainerStarted","Data":"8d498764486bdfa592718beae20ef70fb4b7941f3dcf1a414e1d75ade3702656"}
Nov 02 15:10:11.001109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.001035    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podStartSLOduration=2.001010195 podStartE2EDuration="2.001010195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:10.980136034 +0000 UTC m=+805.234089659" watchObservedRunningTime="2026-11-02 15:10:11.001010195 +0000 UTC m=+805.254963820"
Nov 02 15:10:11.007671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.003971    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:11.007966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.007929    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 15:10:11.637920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.637880    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:11.748674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.746756    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:11.748674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.747706    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:11.748674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.747763    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 15:10:11.998741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.998073    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:11.998741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:11.998137    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 15:10:12.742383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:12.742338    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:12.742680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:12.742628    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 15:10:12.995192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:12.995092    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:12.995473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:12.995444    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 15:10:13.434968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:13.433982    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:13.741969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:13.741878    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:13.741969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:13.741941    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 15:10:13.991672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:13.991019    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:13.991672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:13.991080    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 15:10:14.741723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:14.741694    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:14.742167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:14.742146    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 15:10:15.004435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:15.004343    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:15.012828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:15.012788    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 15:10:15.128941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:15.128891    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:15.742605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:15.742568    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:15.743024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:15.742621    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 15:10:15.991205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:15.991071    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 02 15:10:15.991205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:15.991134    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 02 15:10:16.691385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:16.691314    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:16.744184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:16.742844    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 15:10:16.744184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:16.742906    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 15:10:16.996175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:16.996083    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:16.996514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:16.996489    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:16.997409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:16.997390    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:10:17.746421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:17.746392    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:17.746872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:17.746854    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:17.747568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:17.747543    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:10:18.377282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.377245    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:18.882430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.882391    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"]
Nov 02 15:10:18.887828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.887802    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:18.913888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.913857    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"]
Nov 02 15:10:18.920902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.920875    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tmp-dir\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:18.921044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.920984    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-dshm\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:18.921044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.921015    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tls-certs\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:18.921129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.921077    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-home\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:18.921129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.921114    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564bj\" (UniqueName: \"kubernetes.io/projected/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-kube-api-access-564bj\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:18.921275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.921262    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-model-cache\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:18.923763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.923735    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5113c2\"/\"distinct-2-w6-5113c2-kserve-self-signed-certs\""
Nov 02 15:10:18.964068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:18.964001    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:19.023192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023135    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-model-cache\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.023393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023251    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tmp-dir\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.023469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023394    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-dshm\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.023469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023441    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tls-certs\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.023583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023482    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-home\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.023583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023531    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564bj\" (UniqueName: \"kubernetes.io/projected/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-kube-api-access-564bj\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.023714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023612    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-model-cache\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.023714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.023627    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tmp-dir\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.024032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.024007    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-home\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.026037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.026010    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-dshm\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.026276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.026261    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tls-certs\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.048547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.048519    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564bj\" (UniqueName: \"kubernetes.io/projected/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-kube-api-access-564bj\") pod \"distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") " pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.218589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.217274    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:19.537695 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:10:19.537632    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61f22b8b_7228_4294_9c9b_f74ad0c5e98e.slice/crio-c9b0158df23a652a65fcc232e45a7abb2c9aa52f71a4abddf88ad03d9f2fa391 WatchSource:0}: Error finding container c9b0158df23a652a65fcc232e45a7abb2c9aa52f71a4abddf88ad03d9f2fa391: Status 404 returned error can't find the container with id c9b0158df23a652a65fcc232e45a7abb2c9aa52f71a4abddf88ad03d9f2fa391
Nov 02 15:10:19.537904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.537883    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"]
Nov 02 15:10:19.875195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:19.875162    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:10:20.010380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:20.010342    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" event={"ID":"61f22b8b-7228-4294-9c9b-f74ad0c5e98e","Type":"ContainerStarted","Data":"c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63"}
Nov 02 15:10:20.010380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:20.010382    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" event={"ID":"61f22b8b-7228-4294-9c9b-f74ad0c5e98e","Type":"ContainerStarted","Data":"c9b0158df23a652a65fcc232e45a7abb2c9aa52f71a4abddf88ad03d9f2fa391"}
Nov 02 15:10:20.101958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:20.101905    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podStartSLOduration=2.101888735 podStartE2EDuration="2.101888735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:20.097869813 +0000 UTC m=+814.351823439" watchObservedRunningTime="2026-11-02 15:10:20.101888735 +0000 UTC m=+814.355842360"
Nov 02 15:10:20.221889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:20.221807    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:20.298535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:20.298494    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:22.250336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:22.250299    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:22.708106 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:10:22.708038    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:10:23.146392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:23.146357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:23.997641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:23.997609    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:25.596199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:25.596161    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:27.310583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:27.310492    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:27.459570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:27.459518    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:29.222931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:29.222899    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:29.236183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:29.236155    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:29.281258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:29.281222    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:30.141330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:30.141302    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:30.909567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:30.909522    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:32.592284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:32.592229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:34.244853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:34.244823    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:35.921793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:35.921755    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:37.635169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:37.635132    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:38.871160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:38.871123    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"]
Nov 02 15:10:38.875681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:38.875637    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:38.917320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:38.917284    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"]
Nov 02 15:10:38.949860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:38.949829    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"]
Nov 02 15:10:38.953667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:38.953628    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:38.977456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:38.977422    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-1cc1b2-openshift-default-dockercfg-vsxtg\""
Nov 02 15:10:38.989852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:38.989816    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"]
Nov 02 15:10:39.017591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.017543    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.017881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.017858    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.018053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.018034    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77cbz\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-kube-api-access-77cbz\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.018182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.018167    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.018402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.018385    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.018599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.018585    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.018739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.018721    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.018936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.018914    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.022853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.022819    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.123922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.123830    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.123922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.123883    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.123958    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124085    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124110    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124145    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124187    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124247    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124269    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124456    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124466    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124541    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124569    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124597    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-data\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124610    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77cbz\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-kube-api-access-77cbz\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124621    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istiod-ca-cert\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124688    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2ks\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-kube-api-access-zr2ks\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124700    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-credential-socket\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124717    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124823    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124850    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.124908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124876    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-certs\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.124908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.124889    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.127110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.127082    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-envoy\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.127410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.127389    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-podinfo\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.150271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.150240    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-token\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.150271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.150265    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77cbz\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-kube-api-access-77cbz\") pod \"e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") " pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.194604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.194576    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:39.225945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.225907    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.225972    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226075    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226106    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226173    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2ks\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-kube-api-access-zr2ks\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226250    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226275    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226310    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226310    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226365    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226386    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226548    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.226697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.226637    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.227260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.227232    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.229005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.228981    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.229175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.229158    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.253010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.252976    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2ks\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-kube-api-access-zr2ks\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.253360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.253341    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.272165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.269190    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:39.400856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.397088    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:39.518513 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:10:39.518475    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2f7d2d_c100_4f10_bc92_17e34f8e8527.slice/crio-54035108c3aa456995ef655d6c41da8d8515c9584d1ca605cf495b7ad0491fca WatchSource:0}: Error finding container 54035108c3aa456995ef655d6c41da8d8515c9584d1ca605cf495b7ad0491fca: Status 404 returned error can't find the container with id 54035108c3aa456995ef655d6c41da8d8515c9584d1ca605cf495b7ad0491fca
Nov 02 15:10:39.518665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.518619    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"]
Nov 02 15:10:39.521114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.521078    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.521200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.521148    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.521200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.521178    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.591137 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:10:39.591109    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb5c54c_78b6_44b9_96a3_84b99d088b3c.slice/crio-a24d441d682e2599dc3feeff8ae36b18ec71d3db8301937f400284b3a74109c6 WatchSource:0}: Error finding container a24d441d682e2599dc3feeff8ae36b18ec71d3db8301937f400284b3a74109c6: Status 404 returned error can't find the container with id a24d441d682e2599dc3feeff8ae36b18ec71d3db8301937f400284b3a74109c6
Nov 02 15:10:39.591795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.591774    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"]
Nov 02 15:10:39.594542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.594504    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.594669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.594587    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:39.594730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:39.594704    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:40.205513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.205484    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" event={"ID":"fcb5c54c-78b6-44b9-96a3-84b99d088b3c","Type":"ContainerStarted","Data":"5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab"}
Nov 02 15:10:40.206019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.205537    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" event={"ID":"fcb5c54c-78b6-44b9-96a3-84b99d088b3c","Type":"ContainerStarted","Data":"a24d441d682e2599dc3feeff8ae36b18ec71d3db8301937f400284b3a74109c6"}
Nov 02 15:10:40.207802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.207773    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" event={"ID":"da2f7d2d-c100-4f10-bc92-17e34f8e8527","Type":"ContainerStarted","Data":"ed742df789b3623a2cc88b1b096ceb5db7277db2091d09d13cee48e5346b6164"}
Nov 02 15:10:40.207925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.207825    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" event={"ID":"da2f7d2d-c100-4f10-bc92-17e34f8e8527","Type":"ContainerStarted","Data":"54035108c3aa456995ef655d6c41da8d8515c9584d1ca605cf495b7ad0491fca"}
Nov 02 15:10:40.270873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.270841    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:40.272522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.272496    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:40.272684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.272547    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:40.280764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.280717    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podStartSLOduration=2.280703808 podStartE2EDuration="2.280703808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.267063884 +0000 UTC m=+834.521017574" watchObservedRunningTime="2026-11-02 15:10:40.280703808 +0000 UTC m=+834.534657432"
Nov 02 15:10:40.332532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:40.332479    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podStartSLOduration=2.3324639449999998 podStartE2EDuration="2.332463945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:40.330049469 +0000 UTC m=+834.584003092" watchObservedRunningTime="2026-11-02 15:10:40.332463945 +0000 UTC m=+834.586417570"
Nov 02 15:10:41.139217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:41.139175    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:41.198702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:41.194821    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:41.198702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:41.196692    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:41.198702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:41.196834    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:41.286104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:41.286067    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:41.286689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:41.286638    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:42.212220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:42.211759    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:42.212220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:42.211821    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:42.285673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:42.274071    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:42.285673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:42.274134    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:43.019673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:43.018691    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:43.203672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:43.195068    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:43.203672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:43.195132    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:43.273164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:43.273061    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:43.273452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:43.273420    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:44.131674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:44.129724    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:44.196671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:44.196146    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:44.196671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:44.196229    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:44.286686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:44.283069    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:44.286686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:44.283132    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:45.202686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:45.199623    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:45.202686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:45.199699    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:45.308312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:45.307788    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:45.308312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:45.307861    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:45.905681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:45.879915    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.063801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.063749    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.107674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.107068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:46.143053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.143007    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.327751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.308826    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:46.327751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.308887    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:46.327751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.312063    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.374673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.366828    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.447178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.431716    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.447178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.431823    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:46.447178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.431854    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:46.447178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.446748    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.500670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.489185    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.609745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.589538    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.748674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.748081    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:46.889674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.881181    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:10:46.889674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:46.882174    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:10:47.116778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.101550    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.116778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.110030    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:10:47.159304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.146231    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.159304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.146852    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:47.215713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.194984    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:47.215713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.195045    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:47.280448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.280415    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:47.280854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.280739    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:47.631317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.631269    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.657703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.657515    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.756331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.668548    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.756331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.671526    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:10:47.784816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.779470    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:47.784816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.726519    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:10:47.784816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.780946    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:10:47.800027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.799987    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:10:47.824725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.803364    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:10:47.824725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.803466    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:10:47.849368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:47.849320    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.104792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.081096    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:10:48.150875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.133715    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.153699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.148479    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.223954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.223918    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 15:10:48.244784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.244735    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 15:10:48.297671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.296954    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:10:48.368574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.368532    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:48.368850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.368822    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:48.693425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.693335    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:10:48.716860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.716814    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.738792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.735731    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:10:48.738792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.735792    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.738792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.735734    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:10:48.738792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.736044    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.786668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.778832    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.786668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.780842    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.805100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.804741    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:48.903953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.903026    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:10:48.904817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.904187    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:10:48.917763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.917302    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:10:48.922666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.921694    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:10:48.936879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:48.936853    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:49.054129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:49.053999    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:10:49.066684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:49.066526    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:10:49.272500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:49.272044    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 02 15:10:49.272500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:49.272109    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 02 15:10:49.684002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:49.683002    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:49.684782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:49.684758    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:49.687884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:49.687854    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:10:50.797411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:50.797378    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:51.311890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:51.311840    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:10:51.312120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:51.311920    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:10:51.436886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:51.436849    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:51.636986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:51.636958    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:51.647284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:51.646820    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:51.647850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:51.647829    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:10:52.695880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:52.695845    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:52.708445 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:10:52.708400    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:10:54.000062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.000014    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"]
Nov 02 15:10:54.005302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.005279    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.029641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.029602    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.029826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.029740    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47728e82-309b-40f1-98d4-9cf1955fe7ef-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.029826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.029809    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.029961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.029842    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ljt\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-kube-api-access-x6ljt\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.029961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.029905    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.029961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.029928    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.030116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.029965    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.030116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.030040    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.030116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.030059    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.041809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.041774    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"]
Nov 02 15:10:54.131535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131502    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.131535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131536    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.131776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131619    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.131776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131725    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47728e82-309b-40f1-98d4-9cf1955fe7ef-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.131881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131809    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.131881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131844    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ljt\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-kube-api-access-x6ljt\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.131985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131910    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.131985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131938    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.132095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.131984    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.132151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.132093    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.132151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.132135    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.132255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.132141    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.132373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.132350    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.132547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.132526    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47728e82-309b-40f1-98d4-9cf1955fe7ef-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.134328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.134304    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.134560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.134537    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.163188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.163152    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.165961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.165917    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ljt\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-kube-api-access-x6ljt\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.319626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.319527    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:54.418051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.417928    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:54.645701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.645632    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"]
Nov 02 15:10:54.646000 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:10:54.645978    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47728e82_309b_40f1_98d4_9cf1955fe7ef.slice/crio-a1ced69943b1fee73d903fc186cadb142f8207d3602f95f1469e7d6155724203 WatchSource:0}: Error finding container a1ced69943b1fee73d903fc186cadb142f8207d3602f95f1469e7d6155724203: Status 404 returned error can't find the container with id a1ced69943b1fee73d903fc186cadb142f8207d3602f95f1469e7d6155724203
Nov 02 15:10:54.648675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.648611    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:54.648781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.648706    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:54.648781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:54.648737    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:10:55.107944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:55.107836    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" event={"ID":"47728e82-309b-40f1-98d4-9cf1955fe7ef","Type":"ContainerStarted","Data":"aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e"}
Nov 02 15:10:55.107944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:55.107937    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" event={"ID":"47728e82-309b-40f1-98d4-9cf1955fe7ef","Type":"ContainerStarted","Data":"a1ced69943b1fee73d903fc186cadb142f8207d3602f95f1469e7d6155724203"}
Nov 02 15:10:55.159169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:55.159117    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podStartSLOduration=2.159103106 podStartE2EDuration="2.159103106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:10:55.15683032 +0000 UTC m=+849.410783957" watchObservedRunningTime="2026-11-02 15:10:55.159103106 +0000 UTC m=+849.413056786"
Nov 02 15:10:55.320715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:55.320685    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:10:55.322351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:55.322326    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:10:55.322805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:55.322379    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 15:10:56.088669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:56.087724    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:56.320375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:56.320331    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:10:56.320888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:56.320390    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 15:10:57.320542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:57.320508    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:10:57.321058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:57.320568    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 15:10:57.685897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:57.685863    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:58.320450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:58.320411    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:10:58.320641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:58.320470    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 15:10:59.068675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:59.065245    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:10:59.320884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:59.320801    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:10:59.320884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:59.320865    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 15:10:59.412750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:10:59.410443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:00.320355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:00.320307    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:00.320557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:00.320376    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 15:11:01.043024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:01.042986    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:01.320539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:01.320443    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 15:11:01.320539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:01.320517    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 15:11:02.324966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:02.324933    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:11:02.325346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:02.325315    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:11:02.325974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:02.325953    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:11:02.686191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:02.686146    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:03.650440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.650401    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"]
Nov 02 15:11:03.655875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.655833    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.679145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.679115    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"]
Nov 02 15:11:03.739926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.739890    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.739926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.739923    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.740137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.739958    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.740200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.740178    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.740241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.740227    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.740291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.740277    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.740371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.740357    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.740406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.740385    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.740440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.740433    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfd9\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-kube-api-access-hvfd9\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.841984    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842039    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842057    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842095    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842128    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842155    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfd9\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-kube-api-access-hvfd9\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842219    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842241    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842283    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842550    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-credential-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842593    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-socket\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842696    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-data\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.842782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.842766    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-certs\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.843080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.843058    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istiod-ca-cert\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.844364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.844343    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-envoy\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.844686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.844668    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-podinfo\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.867358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.867326    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-token\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.867941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.867918    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfd9\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-kube-api-access-hvfd9\") pod \"e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") " pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:03.982043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:03.981934    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:04.311970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:04.311934    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"]
Nov 02 15:11:04.313301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:04.313258    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:04.313375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:04.313343    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:04.313418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:04.313386    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:04.412941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:04.412902    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:11:04.506973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:04.506931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:05.280670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:05.269046    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" event={"ID":"fa53c43b-3c4c-47d1-b98c-aca467ff95a0","Type":"ContainerStarted","Data":"b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0"}
Nov 02 15:11:05.280670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:05.269118    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" event={"ID":"fa53c43b-3c4c-47d1-b98c-aca467ff95a0","Type":"ContainerStarted","Data":"10d04d575b2b30b8165ec08376ea0892277fa4bd3ca13e333e2fa9aae6cb4f5b"}
Nov 02 15:11:05.347023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:05.346966    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podStartSLOduration=2.346952174 podStartE2EDuration="2.346952174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:05.333059703 +0000 UTC m=+859.587013320" watchObservedRunningTime="2026-11-02 15:11:05.346952174 +0000 UTC m=+859.600905797"
Nov 02 15:11:05.984053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:05.984019    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:06.019156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:06.019118    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:06.019444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:06.019416    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:06.907672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:06.906405    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:07.015685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:06.998849    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.015685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.010803    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:07.015685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.010885    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:07.171740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.169752    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.171740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.170181    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:11:07.187683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.186774    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.708676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.705098    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.737809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.737748    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.823951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.823897    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.824593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.824561    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 02 15:11:07.849675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.827913    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:07.895940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.885640    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:08.000733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.994897    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:08.000733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:07.995206    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:08.053987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.009132    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.053987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.009596    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:11:08.053987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.026476    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.053987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.026797    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:11:08.179439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.179394    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.180075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.179710    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.185364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.180695    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:11:08.194326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.194302    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:11:08.207704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.194187    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.226672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.209029    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:11:08.364136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.352917    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.707902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.707785    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.770684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.721481    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:11:08.817675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.742693    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.843674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.818665    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:11:08.906672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:08.870214    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.103738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.025362    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:11:09.153674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.100252    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.203674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.100296    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.244836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.244787    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.357758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.304119    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:11:09.367783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.367734    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.406155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.369133    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:09.431840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.431793    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:09.432111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.394345    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.442053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.441962    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:09.633571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.633538    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:11:09.655519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.641496    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:09.774113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.774069    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:09.982166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:09.982064    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.065731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.047965    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.256732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.073334    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.256732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.084200    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:10.256732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.252726    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:10.256732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.223518    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.286673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.199167    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.379671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.374231    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.435796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.435748    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.636962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.636900    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.695786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.695731    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:11:10.893508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.891616    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:10.952524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.952483    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:11.021731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.999214    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:11.021731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:10.999280    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:11.045677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.042742    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:11.060243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.060209    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:11:11.227078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.222799    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:11:11.313680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.311784    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:11.313680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.312609    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:11.341664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.334281    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:11:11.397958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.397892    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:11:11.399164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.399099    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:11.430984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.430960    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:11:11.441670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.431982    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:11:11.441670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.440773    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:11:11.456167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.456040    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:11:11.463695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.462854    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:11:11.998070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.998032    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:11.998356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:11.998328    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:12.717079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:12.717042    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:12.983169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:12.983071    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:12.983464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:12.983436    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:13.983303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:13.983255    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 15:11:13.983853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:13.983333    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 15:11:14.391274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:14.391243    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:16.023829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:16.023783    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:16.024245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:16.023874    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:16.189856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:16.189824    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:16.244306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:16.244272    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:16.244739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:16.244717    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:16.246974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:16.246951    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:11:17.826856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:17.826820    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:19.504788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:19.504756    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:21.236263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:21.236218    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:22.707711 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:11:22.707641    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:11:22.788066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:22.788028    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:22.940114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:22.940073    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:23.888585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:23.888547    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"]
Nov 02 15:11:23.894495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:23.894473    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:23.922967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:23.922939    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"]
Nov 02 15:11:23.942790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:23.942741    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"]
Nov 02 15:11:23.962974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:23.962940    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"]
Nov 02 15:11:23.963154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:23.963063    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.012610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.012573    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.012610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.012612    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.012855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.012629    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.012855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.012777    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.012855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.012811    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxfd\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-kube-api-access-scxfd\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.012972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.012939    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.013054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.013041    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.013151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.013139    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.013198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.013177    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.114347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114295    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.114347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114346    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.114600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114437    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q85gz\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-kube-api-access-q85gz\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.114600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114490    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.114600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114550    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.114600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114582    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.114920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114735    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.114920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114816    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.114920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114859    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.115027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114930    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114936    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.114996    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115025    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115057    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115088    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxfd\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-kube-api-access-scxfd\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115138    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115172    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.115228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115211    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.115536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115332    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115603    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115668    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115702    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.115913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.115744    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.117598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.117573    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.117940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.117923    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.139711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.139622    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.139979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.139963    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxfd\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-kube-api-access-scxfd\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.211581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.211541    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:24.216380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216263    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216295    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216324    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q85gz\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-kube-api-access-q85gz\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216370    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216421    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216468    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216517    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216614    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216662    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.216991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.216967    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.217148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.217126    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.217227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.217194    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.217298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.217276    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.217543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.217518    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.219041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.219015    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.219331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.219312    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.242544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.242516    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q85gz\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-kube-api-access-q85gz\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.242738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.242516    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.281756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.281619    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:24.417671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.416376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:24.451420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.451386    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"]
Nov 02 15:11:24.478678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.478145    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.480406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.480376    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"]
Nov 02 15:11:24.555368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.555342    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"]
Nov 02 15:11:24.556472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.556437    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.556570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.556516    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.556570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.556561    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.630021 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:11:24.629973    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3da149_f1cd_4386_9d8e_8bd2ec2122f1.slice/crio-2092244028d8d42d13535fbedb8ac8971512d830bc6254bd9718e9e2bdcc0814 WatchSource:0}: Error finding container 2092244028d8d42d13535fbedb8ac8971512d830bc6254bd9718e9e2bdcc0814: Status 404 returned error can't find the container with id 2092244028d8d42d13535fbedb8ac8971512d830bc6254bd9718e9e2bdcc0814
Nov 02 15:11:24.631450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.631426    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"]
Nov 02 15:11:24.632871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.632835    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.632977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.632916    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.632977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.632967    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:24.635592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.635558    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.635690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.635639    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.635753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.635724    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.635801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.635768    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.635925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.635903    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.636122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.636097    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.636177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.636160    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw245\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-kube-api-access-fw245\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.636242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.636226    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.636304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.636288    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.721641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.721361    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" event={"ID":"bbd3322e-04e9-4d6d-98ad-e77ca340566a","Type":"ContainerStarted","Data":"5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4"}
Nov 02 15:11:24.721641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.721425    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" event={"ID":"bbd3322e-04e9-4d6d-98ad-e77ca340566a","Type":"ContainerStarted","Data":"af4c92607dded5fa11d9e88d78e4cf12907f30f17c81becf31582fe1890ab537"}
Nov 02 15:11:24.723510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.723477    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" event={"ID":"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1","Type":"ContainerStarted","Data":"7eb6cdf4f1ecbc09a72f03e7129080d5e4fc2b2eccdeff88a6d336670730caec"}
Nov 02 15:11:24.723660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.723543    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" event={"ID":"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1","Type":"ContainerStarted","Data":"2092244028d8d42d13535fbedb8ac8971512d830bc6254bd9718e9e2bdcc0814"}
Nov 02 15:11:24.740720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.740679    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.740905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.740791    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.740905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.740844    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.741021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.740967    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.741074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.741031    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.741144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.741085    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw245\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-kube-api-access-fw245\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.741144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.741118    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.741248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.741156    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.741248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.741218    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.750182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.749481    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-data\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.750182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.749800    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-envoy\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.750182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.749931    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-podinfo\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.750182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.750052    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-certs\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.750182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.750178    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.750457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.750267    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-credential-socket\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.750601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.750582    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istiod-ca-cert\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.751001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.750980    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"]
Nov 02 15:11:24.756502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.756482    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.782750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.781083    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw245\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-kube-api-access-fw245\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.782750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.781134    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"]
Nov 02 15:11:24.786309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.786277    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-token\") pod \"e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") " pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.791711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.787057    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podStartSLOduration=1.7870408599999998 podStartE2EDuration="1.78704086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:24.782188967 +0000 UTC m=+879.036142592" watchObservedRunningTime="2026-11-02 15:11:24.78704086 +0000 UTC m=+879.040994485"
Nov 02 15:11:24.816737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.816332    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:24.846026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.845987    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846059    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846171    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d8223-7b91-4d72-9073-bfc6afa6888c-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846231    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846255    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846374    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846426    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846496    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h59lt\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-kube-api-access-h59lt\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.846596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.846549    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.923904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.923840    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podStartSLOduration=1.9238165440000001 podStartE2EDuration="1.923816544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:24.903155607 +0000 UTC m=+879.157109230" watchObservedRunningTime="2026-11-02 15:11:24.923816544 +0000 UTC m=+879.177770174"
Nov 02 15:11:24.949104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949060    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.949270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949160    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h59lt\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-kube-api-access-h59lt\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.949270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949213    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.949391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949359    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.949451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949410    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.950760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949542    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d8223-7b91-4d72-9073-bfc6afa6888c-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.950760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949593    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.950760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949619    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.950760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.949740    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.966983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.962245    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.966983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.962882    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.966983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.963129    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.969740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.967771    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.969740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.968609    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d8223-7b91-4d72-9073-bfc6afa6888c-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.969740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.969395    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.969740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.969624    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.994417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.994385    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h59lt\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-kube-api-access-h59lt\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:24.999977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:24.999920    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:25.092779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.092748    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:25.197270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.187622    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"]
Nov 02 15:11:25.214093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.213714    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:25.223435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.215204    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:25.223435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.215272    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:25.226082 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:11:25.226050    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29800bd_eb81_4d8b_8561_51eb0c07cc2b.slice/crio-ba312c305f3f02e89a2c187907b4e0fc90d2c575034b22439b16b2d0d405ec84 WatchSource:0}: Error finding container ba312c305f3f02e89a2c187907b4e0fc90d2c575034b22439b16b2d0d405ec84: Status 404 returned error can't find the container with id ba312c305f3f02e89a2c187907b4e0fc90d2c575034b22439b16b2d0d405ec84
Nov 02 15:11:25.229429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.229392    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.229591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.229577    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.229720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.229695    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.305161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.305116    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:25.323379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.323343    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:25.323522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.323404    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:25.501775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.501736    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"]
Nov 02 15:11:25.542151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.541327    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.542151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.541417    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.542151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.541458    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:25.756024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.755990    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" event={"ID":"e94d8223-7b91-4d72-9073-bfc6afa6888c","Type":"ContainerStarted","Data":"489e3234b928ffab662053d4b227c90b5da13f1d473c7661e038c555fca1d58d"}
Nov 02 15:11:25.758742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.758715    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" event={"ID":"f29800bd-eb81-4d8b-8561-51eb0c07cc2b","Type":"ContainerStarted","Data":"9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea"}
Nov 02 15:11:25.758981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.758957    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" event={"ID":"f29800bd-eb81-4d8b-8561-51eb0c07cc2b","Type":"ContainerStarted","Data":"ba312c305f3f02e89a2c187907b4e0fc90d2c575034b22439b16b2d0d405ec84"}
Nov 02 15:11:25.834111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.834071    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:25.852733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:25.852671    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podStartSLOduration=1.852635287 podStartE2EDuration="1.852635287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:25.833936269 +0000 UTC m=+880.087889894" watchObservedRunningTime="2026-11-02 15:11:25.852635287 +0000 UTC m=+880.106588912"
Nov 02 15:11:26.216046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.216008    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.216757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.216719    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:26.346991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.346952    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.347287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.347256    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:26.479601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.478839    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:26.836737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.828726    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:26.836737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.828819    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:26.872828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:26.862744    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" event={"ID":"e94d8223-7b91-4d72-9073-bfc6afa6888c","Type":"ContainerStarted","Data":"9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da"}
Nov 02 15:11:27.096728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.095995    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:27.108023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.107984    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.108295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.108262    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:27.221482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.221446    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.222052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.222021    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:27.295670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.292258    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.295670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.292318    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:27.459791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.459739    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:27.683853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.683802    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:27.823735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.821053    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:27.823735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:27.821117    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:28.132674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.127358    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.132674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.127425    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:28.219190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.219152    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.219503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.219460    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:28.300673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.292085    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.300673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.292161    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:28.325674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.324492    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:28.367674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.364728    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:28.391730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.391097    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:28.432697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.432003    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:28.462686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.456731    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:11:28.545674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.535773    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:28.545674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.536398    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:11:28.545674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.540504    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:28.613669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.611823    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podStartSLOduration=4.611797949 podStartE2EDuration="4.611797949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:26.933021954 +0000 UTC m=+881.186975579" watchObservedRunningTime="2026-11-02 15:11:28.611797949 +0000 UTC m=+882.865751573"
Nov 02 15:11:28.725734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.704732    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:28.725734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.705183    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:11:28.725734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.715751    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:28.821669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.816017    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:11:28.821669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.817849    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:28.821669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:28.817892    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:29.244894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.240793    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 15:11:29.244894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.243062    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.244894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.243112    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:29.255702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.250756    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:11:29.318689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.289106    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.318689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.289231    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:29.318689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.290796    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.318689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.290836    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:29.334932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.334871    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:29.378700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.343671    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:11:29.430606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.395000    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:29.430606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.402201    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 02 15:11:29.430606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.402373    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:11:29.444468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.432883    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:29.444855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.444828    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:11:29.476370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.474897    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:29.476370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.475831    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:11:29.686481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.686429    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:29.737682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.727724    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:11:29.820260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.819037    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:29.820260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.819116    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:29.820260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.819193    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:29.828066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:29.824044    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:11:30.123736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.100482    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.123736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.100549    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:30.273672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.273354    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.273672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.273410    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:30.273672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.273488    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:30.297812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.274230    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:30.297812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.287602    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:30.297812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.287741    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:11:30.313417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.311967    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.313417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.312025    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:30.343162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.343116    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:30.541671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.540738    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:30.541671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.541155    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:30.566668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.546337    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:11:30.715211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.715020    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:30.742282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.741887    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:30.821809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.821760    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:11:30.839539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.839435    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:30.839852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.839512    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:30.985606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:30.985513    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:31.117749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.117711    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.118022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.117996    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:31.217634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.217513    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.217634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.217572    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:31.290735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.273600    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:31.291179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.291145    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:11:31.308704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.301762    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:31.308704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.302189    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.308704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.302234    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:31.370883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.370839    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:31.575827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.567114    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:31.575827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.567829    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:31.575827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.568111    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:31.706923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.705935    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:11:31.716482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.716331    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:31.743081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.742268    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:11:31.829863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.829747    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:31.834815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.834780    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:31.932840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:31.932712    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:32.109304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.108927    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.109304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.108987    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:32.260692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.240131    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.260692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.240203    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:32.276673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.272724    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:11:32.307674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.294160    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:32.318733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.302152    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.319078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.319043    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:32.375166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.375126    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:32.577673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.576877    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:32.605671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.578746    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:32.605671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.580092    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:11:32.862573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.862269    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:32.862573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.862331    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:32.862838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.862754    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:32.903838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:32.903781    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:33.069517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.069473    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:11:33.134638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.134547    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:33.134934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.134902    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:33.173729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.169597    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:11:33.323291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.268773    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 15:11:33.323291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.268849    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 15:11:33.323291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.277017    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:33.323291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.319894    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:33.379717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.368670    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:33.379717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.368760    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:33.674680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.665012    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:11:33.840026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.682633    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:33.840807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.751764    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:33.841521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.755765    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:33.841981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.819719    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:33.842177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:33.842136    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:34.016060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.015949    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:34.237669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.229272    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:34.239611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.239575    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:34.239853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.239809    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:34.293444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.293239    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 15:11:34.293444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.293309    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 15:11:34.322708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.322678    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:11:34.334935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.334822    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:34.386603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.386558    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:11:34.520596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.503410    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:11:34.525079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.523971    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:11:34.532972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.532942    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:11:34.560528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.560440    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:34.578691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.562079    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:34.589052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.589022    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:11:34.836675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.834817    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 02 15:11:34.836675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:34.834882    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 02 15:11:35.093360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.093252    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 15:11:35.093602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.093563    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 15:11:35.169860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.169823    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:35.314139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.309980    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:35.314139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.311740    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:35.333899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.332918    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:11:35.355968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.355879    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"]
Nov 02 15:11:35.373846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.373824    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.374163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.374144    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"]
Nov 02 15:11:35.390620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.390516    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:11:35.390620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.390589    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:35.425696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.425641    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.425960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.425934    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwkf\" (UniqueName: \"kubernetes.io/projected/a6497238-fc86-419e-b767-94bb8618ff96-kube-api-access-ddwkf\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.426088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.426058    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.527992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.527630    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.528351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.528324    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.538688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.534382    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwkf\" (UniqueName: \"kubernetes.io/projected/a6497238-fc86-419e-b767-94bb8618ff96-kube-api-access-ddwkf\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.538688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.529180    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.538688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.535955    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.578675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.576426    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwkf\" (UniqueName: \"kubernetes.io/projected/a6497238-fc86-419e-b767-94bb8618ff96-kube-api-access-ddwkf\") pod \"maas-api-revoke-sub-65c3b254-zjrgv\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:35.697671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:35.695169    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:36.025180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.025140    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"]
Nov 02 15:11:36.025771 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:11:36.025740    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6497238_fc86_419e_b767_94bb8618ff96.slice/crio-b675da5866d9e229fea30f6ef46f0ec41c4c4d153f65ffacecf379e9a473b92e WatchSource:0}: Error finding container b675da5866d9e229fea30f6ef46f0ec41c4c4d153f65ffacecf379e9a473b92e: Status 404 returned error can't find the container with id b675da5866d9e229fea30f6ef46f0ec41c4c4d153f65ffacecf379e9a473b92e
Nov 02 15:11:36.028210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.028190    7019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:11:36.115918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.114292    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:36.115918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.115904    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:36.118517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.118489    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:11:36.158174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.158142    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:36.158876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.158853    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:36.160211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.160191    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:11:36.883333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.883289    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv" event={"ID":"a6497238-fc86-419e-b767-94bb8618ff96","Type":"ContainerStarted","Data":"b675da5866d9e229fea30f6ef46f0ec41c4c4d153f65ffacecf379e9a473b92e"}
Nov 02 15:11:36.992329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:36.992285    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:38.918077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:38.917974    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:39.919884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:39.919851    7019 generic.go:356] "Generic (PLEG): container finished" podID="a6497238-fc86-419e-b767-94bb8618ff96" containerID="b148a5bfa87dcb1fef9b18246eba96a8b891fd11c8c1c11e6820e282a191bc53" exitCode=0
Nov 02 15:11:39.920266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:39.919933    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv" event={"ID":"a6497238-fc86-419e-b767-94bb8618ff96","Type":"ContainerDied","Data":"b148a5bfa87dcb1fef9b18246eba96a8b891fd11c8c1c11e6820e282a191bc53"}
Nov 02 15:11:40.619767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:40.619722    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:42.392708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.380399    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.574197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.573478    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 02 15:11:42.574197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.573826    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.578873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.575701    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.719823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.717957    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.750665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.748039    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.750665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.750122    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:42.926839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.926780    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:42.971762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.964089    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6497238-fc86-419e-b767-94bb8618ff96-kube-api-access-ddwkf\" (UniqueName: \"kubernetes.io/projected/a6497238-fc86-419e-b767-94bb8618ff96-kube-api-access-ddwkf\") pod \"a6497238-fc86-419e-b767-94bb8618ff96\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") "
Nov 02 15:11:42.977688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.976534    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-xks-ca\") pod \"a6497238-fc86-419e-b767-94bb8618ff96\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") "
Nov 02 15:11:42.977688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:42.976630    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-openshift-ca\") pod \"a6497238-fc86-419e-b767-94bb8618ff96\" (UID: \"a6497238-fc86-419e-b767-94bb8618ff96\") "
Nov 02 15:11:43.082789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.082726    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6497238-fc86-419e-b767-94bb8618ff96-kube-api-access-ddwkf" pod "a6497238-fc86-419e-b767-94bb8618ff96" (UID: "a6497238-fc86-419e-b767-94bb8618ff96"). InnerVolumeSpecName "kube-api-access-ddwkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:43.084342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.084311    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-openshift-ca" pod "a6497238-fc86-419e-b767-94bb8618ff96" (UID: "a6497238-fc86-419e-b767-94bb8618ff96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:43.171674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.170849    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-xks-ca" pod "a6497238-fc86-419e-b767-94bb8618ff96" (UID: "a6497238-fc86-419e-b767-94bb8618ff96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:43.198685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.196762    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.304736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.278875    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:11:43.305006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.304975    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwkf\" (UniqueName: \"kubernetes.io/projected/a6497238-fc86-419e-b767-94bb8618ff96-kube-api-access-ddwkf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:11:43.305123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.305108    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6497238-fc86-419e-b767-94bb8618ff96-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:11:43.329881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.328809    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.390242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.390188    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.391679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.391633    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:11:43.580991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.580890    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.581939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.581909    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.582747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.582728    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:11:43.584160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.583754    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.584598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.584583    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:11:43.722749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.722701    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:43.724392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.724365    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:11:43.766925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.755162    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:43.766925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.755696    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:11:43.788708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.784829    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv" event={"ID":"a6497238-fc86-419e-b767-94bb8618ff96","Type":"ContainerDied","Data":"b675da5866d9e229fea30f6ef46f0ec41c4c4d153f65ffacecf379e9a473b92e"}
Nov 02 15:11:43.792789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.792753    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b675da5866d9e229fea30f6ef46f0ec41c4c4d153f65ffacecf379e9a473b92e"
Nov 02 15:11:43.793100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.793081    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"
Nov 02 15:11:43.943931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:43.943894    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:11:44.211391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.211259    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.337825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.337773    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:44.365674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.340523    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 02 15:11:44.398964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.398908    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.438705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.438503    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:11:44.438705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.438591    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:11:44.438705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.438636    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:11:44.587899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.583005    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:44.587899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.583386    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:11:44.614246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.607778    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:11:44.614246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.609298    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.770191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.768856    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.770191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.769402    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:44.800692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.788109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:44.945616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:44.945574    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.214729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.213821    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.386691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.338327    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.491728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.428257    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.589710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.586735    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.589710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.587173    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:11:45.632673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.610325    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.773946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.773809    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.775799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.775073    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:45.953128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:45.953078    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.522671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.514033    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.544630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.544584    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.562107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.562060    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Nov 02 15:11:46.563097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.563065    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.668747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.665368    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:11:46.669477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.669436    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.688565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.688506    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.784364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.784256    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:46.808674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:46.788327    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.018016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.017966    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.198026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.197988    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:47.201815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.201771    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.404977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.404926    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:47.405271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.405227    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.559606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.559497    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:47.582692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.582625    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.609804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.587149    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.609804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.587279    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.661703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.637768    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:47.661703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.637831    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.690078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.690019    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.692859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.692817    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.727789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.727721    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.774665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.774616    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:47.774960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.774917    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:47.806800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.806756    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:47.835303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.835205    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.146592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:47.954790    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.159686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.018677    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.183672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.042826    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:48.183672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.162155    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.226170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.226136    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:48.226435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.226406    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.333243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.333200    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 02 15:11:48.403552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.403428    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:48.403870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.403823    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.454871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.454835    7019 patch_prober.go:29] interesting pod/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:48.455225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.455191    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.620346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.612349    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.620346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.612744    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.627112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.627067    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.691030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.690920    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.697773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.697730    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.724205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.724161    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.747597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.746761    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:11:48.808894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.808802    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:48.954948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:48.954812    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.147367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.147312    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.169793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.169746    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.179062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.179030    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:49.406933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.406885    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 02 15:11:49.624671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.612841    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.629492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.629453    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.644670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.627769    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:11:49.750806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.735067    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.750806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.735836    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.750806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.749057    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:49.811040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.810989    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:11:49.917702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.917586    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:11:49.940342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.938087    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:11:49.960332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.959982    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:11:49.977031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.964951    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:11:49.980428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.980148    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:11:49.980998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.980963    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:11:49.990866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.990158    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:11:49.995671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:49.992423    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:11:50.031082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:50.030998    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:11:51.106840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:51.106800    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:52.010571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.010541    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"]
Nov 02 15:11:52.011324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.011301    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a6497238-fc86-419e-b767-94bb8618ff96" containerName="revoke-keys"
Nov 02 15:11:52.011324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.011321    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6497238-fc86-419e-b767-94bb8618ff96" containerName="revoke-keys"
Nov 02 15:11:52.011538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.011436    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6497238-fc86-419e-b767-94bb8618ff96" containerName="revoke-keys"
Nov 02 15:11:52.016212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.016191    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.036074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.036051    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"]
Nov 02 15:11:52.046097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.046067    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:11:52.046245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.046066    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:11:52.100488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.100443    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.100719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.100516    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.100825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.100796    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszt6\" (UniqueName: \"kubernetes.io/projected/652c69ed-2903-4c5e-a33b-71458c773222-kube-api-access-cszt6\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.201750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.201715    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.202253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.201793    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.202253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.201953    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszt6\" (UniqueName: \"kubernetes.io/projected/652c69ed-2903-4c5e-a33b-71458c773222-kube-api-access-cszt6\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.202618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.202594    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.204364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.204337    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.227407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.227375    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszt6\" (UniqueName: \"kubernetes.io/projected/652c69ed-2903-4c5e-a33b-71458c773222-kube-api-access-cszt6\") pod \"maas-api-revoke-sub-c2d95328-q8qbr\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.331117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.331035    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:52.640748 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:11:52.640714    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652c69ed_2903_4c5e_a33b_71458c773222.slice/crio-c261a184033ec41b7304e3eace75fc8cb9283be7d0dddedb84f3d47838d5d079 WatchSource:0}: Error finding container c261a184033ec41b7304e3eace75fc8cb9283be7d0dddedb84f3d47838d5d079: Status 404 returned error can't find the container with id c261a184033ec41b7304e3eace75fc8cb9283be7d0dddedb84f3d47838d5d079
Nov 02 15:11:52.641230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.641207    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"]
Nov 02 15:11:52.718304 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:11:52.718255    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:11:52.797498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:52.797454    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:53.321685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:53.321588    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr" event={"ID":"652c69ed-2903-4c5e-a33b-71458c773222","Type":"ContainerStarted","Data":"f79e7190718c173f7f308a86caef602645cd6f0b8fba33e4bb3f32dfbc042bdd"}
Nov 02 15:11:53.322059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:53.321698    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr" event={"ID":"652c69ed-2903-4c5e-a33b-71458c773222","Type":"ContainerStarted","Data":"c261a184033ec41b7304e3eace75fc8cb9283be7d0dddedb84f3d47838d5d079"}
Nov 02 15:11:53.368872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:53.368819    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr" podStartSLOduration=2.368802973 podStartE2EDuration="2.368802973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:53.365182425 +0000 UTC m=+907.619136049" watchObservedRunningTime="2026-11-02 15:11:53.368802973 +0000 UTC m=+907.622756596"
Nov 02 15:11:54.193116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.193081    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:54.212414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.212370    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"]
Nov 02 15:11:54.216968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.216941    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.249565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.249533    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"]
Nov 02 15:11:54.330834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.330794    7019 generic.go:356] "Generic (PLEG): container finished" podID="652c69ed-2903-4c5e-a33b-71458c773222" containerID="f79e7190718c173f7f308a86caef602645cd6f0b8fba33e4bb3f32dfbc042bdd" exitCode=0
Nov 02 15:11:54.331336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.330922    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr" event={"ID":"652c69ed-2903-4c5e-a33b-71458c773222","Type":"ContainerDied","Data":"f79e7190718c173f7f308a86caef602645cd6f0b8fba33e4bb3f32dfbc042bdd"}
Nov 02 15:11:54.345892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.345859    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.345931    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmkg\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-kube-api-access-rtmkg\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.346091    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.346133    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a555b59-c7fc-495e-ab24-1365829e3293-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.346231    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.346260    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.346307    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a555b59-c7fc-495e-ab24-1365829e3293-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.346373    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.346446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.346432    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.430658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.427256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.448666    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.448794    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.448870    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmkg\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-kube-api-access-rtmkg\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.448966    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.448994    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a555b59-c7fc-495e-ab24-1365829e3293-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.449052    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.449082    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.449153    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a555b59-c7fc-495e-ab24-1365829e3293-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.449215    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.450017    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.450259    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.450769    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.451142    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.453232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.451920    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a555b59-c7fc-495e-ab24-1365829e3293-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.465914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.464817    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.469266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.469233    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a555b59-c7fc-495e-ab24-1365829e3293-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.476328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.476297    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.476743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.476627    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmkg\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-kube-api-access-rtmkg\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.533297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.533258    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:54.862549 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:11:54.862516    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a555b59_c7fc_495e_ab24_1365829e3293.slice/crio-edeb38912a293192e47ca7cf6e739a0c69be7af6113f9ced4c7c040d69eca3d4 WatchSource:0}: Error finding container edeb38912a293192e47ca7cf6e739a0c69be7af6113f9ced4c7c040d69eca3d4: Status 404 returned error can't find the container with id edeb38912a293192e47ca7cf6e739a0c69be7af6113f9ced4c7c040d69eca3d4
Nov 02 15:11:54.862921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.862889    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"]
Nov 02 15:11:54.864633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.864596    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:54.864785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.864717    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:54.864785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:54.864770    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:11:55.345682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.345545    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" event={"ID":"3a555b59-c7fc-495e-ab24-1365829e3293","Type":"ContainerStarted","Data":"874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7"}
Nov 02 15:11:55.345682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.345611    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" event={"ID":"3a555b59-c7fc-495e-ab24-1365829e3293","Type":"ContainerStarted","Data":"edeb38912a293192e47ca7cf6e739a0c69be7af6113f9ced4c7c040d69eca3d4"}
Nov 02 15:11:55.400567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.400496    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podStartSLOduration=1.400476546 podStartE2EDuration="1.400476546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:55.399187654 +0000 UTC m=+909.653141279" watchObservedRunningTime="2026-11-02 15:11:55.400476546 +0000 UTC m=+909.654430170"
Nov 02 15:11:55.534446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.534416    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:11:55.535961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.535935    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 15:11:55.536152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.535984    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 15:11:55.560205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.560173    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:55.685670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.669208    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/652c69ed-2903-4c5e-a33b-71458c773222-kube-api-access-cszt6\" (UniqueName: \"kubernetes.io/projected/652c69ed-2903-4c5e-a33b-71458c773222-kube-api-access-cszt6\") pod \"652c69ed-2903-4c5e-a33b-71458c773222\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") "
Nov 02 15:11:55.685670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.669283    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-xks-ca\") pod \"652c69ed-2903-4c5e-a33b-71458c773222\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") "
Nov 02 15:11:55.685670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.669596    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-openshift-ca\") pod \"652c69ed-2903-4c5e-a33b-71458c773222\" (UID: \"652c69ed-2903-4c5e-a33b-71458c773222\") "
Nov 02 15:11:55.685670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.670691    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-openshift-ca" pod "652c69ed-2903-4c5e-a33b-71458c773222" (UID: "652c69ed-2903-4c5e-a33b-71458c773222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:11:55.685670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.676614    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652c69ed-2903-4c5e-a33b-71458c773222-kube-api-access-cszt6" pod "652c69ed-2903-4c5e-a33b-71458c773222" (UID: "652c69ed-2903-4c5e-a33b-71458c773222"). InnerVolumeSpecName "kube-api-access-cszt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:11:55.695889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.692301    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-xks-ca" pod "652c69ed-2903-4c5e-a33b-71458c773222" (UID: "652c69ed-2903-4c5e-a33b-71458c773222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:11:55.787671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.787199    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:11:55.787671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.787242    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cszt6\" (UniqueName: \"kubernetes.io/projected/652c69ed-2903-4c5e-a33b-71458c773222-kube-api-access-cszt6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:11:55.787671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:55.787260    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/652c69ed-2903-4c5e-a33b-71458c773222-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:11:56.201672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:56.201615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:56.355195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:56.355161    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"
Nov 02 15:11:56.355688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:56.355627    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr" event={"ID":"652c69ed-2903-4c5e-a33b-71458c773222","Type":"ContainerDied","Data":"c261a184033ec41b7304e3eace75fc8cb9283be7d0dddedb84f3d47838d5d079"}
Nov 02 15:11:56.355798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:56.355740    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c261a184033ec41b7304e3eace75fc8cb9283be7d0dddedb84f3d47838d5d079"
Nov 02 15:11:56.534304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:56.534223    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 15:11:56.534304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:56.534277    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 15:11:57.533629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:57.533595    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 15:11:57.534088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:57.533688    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 15:11:58.056274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:58.056239    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:58.533725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:58.533687    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 15:11:58.534108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:58.533762    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 15:11:59.251807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:59.251775    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:11:59.533781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:59.533694    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 15:11:59.533781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:59.533754    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 15:11:59.931420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:11:59.931376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:00.534103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:00.534064    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 15:12:00.534501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:00.534124    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 15:12:01.533602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:01.533565    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 02 15:12:01.533973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:01.533933    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 02 15:12:01.613056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:01.613019    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:02.538851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:02.538821    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:12:02.539334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:02.539306    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:12:02.540126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:02.540106    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:12:03.403293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:03.403249    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:05.264837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:05.264800    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:07.041437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:07.041402    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:08.645756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:08.645722    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:09.053963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.053880    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"]
Nov 02 15:12:09.054505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.054491    7019 cpu_manager.go:414] "containerMap: removing container" podUID="652c69ed-2903-4c5e-a33b-71458c773222" containerName="revoke-keys"
Nov 02 15:12:09.054505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.054505    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="652c69ed-2903-4c5e-a33b-71458c773222" containerName="revoke-keys"
Nov 02 15:12:09.054677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.054634    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="652c69ed-2903-4c5e-a33b-71458c773222" containerName="revoke-keys"
Nov 02 15:12:09.057438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.057422    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.096536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.096505    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"]
Nov 02 15:12:09.219846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.219807    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.219925    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.219984    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.220028    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.220044    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.220268    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvtpn\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-kube-api-access-mvtpn\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.220312    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.220393    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.220471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.220460    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.321980    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322089    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322134    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322158    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322219    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322243    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322417    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvtpn\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-kube-api-access-mvtpn\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322447    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322480    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322599    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-certs\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322761    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.322960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322934    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-credential-socket\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.323000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322960    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-data\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.323000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.322973    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istiod-ca-cert\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.324509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.324482    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-envoy\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.324834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.324813    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-podinfo\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.347785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.347758    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvtpn\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-kube-api-access-mvtpn\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.348978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.348959    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-token\") pod \"e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") " pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.390641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.390597    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:09.713333 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:12:09.713301    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeed7de_92f1_4222_b9cd_e7edff50e02c.slice/crio-f4b827ad9b3bc7276319ee98179fbc13b47be60c0d8e766e41ef85f7c76bd458 WatchSource:0}: Error finding container f4b827ad9b3bc7276319ee98179fbc13b47be60c0d8e766e41ef85f7c76bd458: Status 404 returned error can't find the container with id f4b827ad9b3bc7276319ee98179fbc13b47be60c0d8e766e41ef85f7c76bd458
Nov 02 15:12:09.713759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.713738    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"]
Nov 02 15:12:09.716180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.716143    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:09.716280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.716222    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:09.716358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.716265    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:09.890217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.890167    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"]
Nov 02 15:12:09.895513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.895486    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:09.925204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.925164    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"]
Nov 02 15:12:09.984867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:09.984764    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:10.033099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033060    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhhbf\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-kube-api-access-zhhbf\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.033276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033223    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.033446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033426    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.033533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033463    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94546aca-dcfe-441c-a14c-16735eb4bc00-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.033697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033640    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.033810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033785    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.033942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033867    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.034005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.033985    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.034055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.034027    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.135567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.135530    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.135745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.135603    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.135745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.135713    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.135825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.135786    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.135825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.135814    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.135915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.135844    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhhbf\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-kube-api-access-zhhbf\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.135965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.135947    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.136047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.136008    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-data\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.136047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.136026    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-credential-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.136148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.136120    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.136148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.136142    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94546aca-dcfe-441c-a14c-16735eb4bc00-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.136451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.136425    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-certs\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.136576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.136472    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-socket\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.137003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.136982    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94546aca-dcfe-441c-a14c-16735eb4bc00-istiod-ca-cert\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.139036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.139013    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-envoy\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.139243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.139225    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-podinfo\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.161015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.160987    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-token\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.161142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.161043    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhhbf\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-kube-api-access-zhhbf\") pod \"e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") " pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.228980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.228927    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:10.498720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.497632    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:10.505093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.505059    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" event={"ID":"bbeed7de-92f1-4222-b9cd-e7edff50e02c","Type":"ContainerStarted","Data":"f0ad15fd25efb1c7e79b61d609e09c2923e105af4c0a0a8a5fd55a8a00774df1"}
Nov 02 15:12:10.505264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.505117    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" event={"ID":"bbeed7de-92f1-4222-b9cd-e7edff50e02c","Type":"ContainerStarted","Data":"f4b827ad9b3bc7276319ee98179fbc13b47be60c0d8e766e41ef85f7c76bd458"}
Nov 02 15:12:10.578673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.572157    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podStartSLOduration=1.5721317190000001 podStartE2EDuration="1.572131719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:10.557400041 +0000 UTC m=+924.811353668" watchObservedRunningTime="2026-11-02 15:12:10.572131719 +0000 UTC m=+924.826085670"
Nov 02 15:12:10.604247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.594447    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"]
Nov 02 15:12:10.614876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.610852    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:10.614876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.610952    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:10.614876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:10.610995    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:11.392402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:11.392370    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:11.393075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:11.393051    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:11.393231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:11.393206    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:11.517765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:11.517729    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" event={"ID":"94546aca-dcfe-441c-a14c-16735eb4bc00","Type":"ContainerStarted","Data":"3d32dbd7d6d6df1cbeb2029592e7614bc528c49487b0e3cdab58ef9d69564322"}
Nov 02 15:12:11.517942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:11.517785    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" event={"ID":"94546aca-dcfe-441c-a14c-16735eb4bc00","Type":"ContainerStarted","Data":"d18aeafeaf044e86c4d2950585f66fea46cb329f77b1238c8d14ae8aae9b3e3f"}
Nov 02 15:12:11.599955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:11.599817    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podStartSLOduration=2.59979908 podStartE2EDuration="2.59979908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:11.599201557 +0000 UTC m=+925.853155210" watchObservedRunningTime="2026-11-02 15:12:11.59979908 +0000 UTC m=+925.853752705"
Nov 02 15:12:12.230433 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:12.230387    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:12.231545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:12.231505    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:12.231713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:12.231571    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:12.391184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:12.391131    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:12.391502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:12.391455    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:12.419181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:12.419150    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:13.230207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:13.230169    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:13.230394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:13.230246    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:13.395206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:13.395164    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:13.395674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:13.395607    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:13.640494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:13.640459    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:12:14.194338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:14.194291    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:14.230087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:14.230042    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:14.230294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:14.230109    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:14.391662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:14.391608    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:14.391936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:14.391907    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:15.244669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:15.237862    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:15.244669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:15.237921    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:15.432683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:15.429798    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:15.432683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:15.429863    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:15.646735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:15.630877    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:15.789580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:15.789533    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:15.804001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:15.799810    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:16.055095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.052676    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:16.262670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.247305    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:16.262670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.247375    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:16.401580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.401545    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:16.402320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.402290    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:16.539836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.539782    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:12:16.617673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.593068    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:12:16.638272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.637967    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:16.659042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.648273    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.659042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.648563    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:12:16.659042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.649993    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:12:16.713302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.713263    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.752777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.752741    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:12:16.801827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.801781    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:16.802458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.802433    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:12:16.803543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.803498    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:16.968477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:16.968386    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.313988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.313890    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:17.366941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.366894    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:17.401407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.401371    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:17.401711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.401681    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:17.647671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.643801    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.647671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.644276    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:17.647671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.645483    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:12:17.647671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.645540    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:12:17.755345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.753717    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:12:17.755345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.754386    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:12:17.755345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.754733    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.006723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.973586    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.006723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.975231    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.006723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.978881    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Nov 02 15:12:18.006723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:17.979231    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.235850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.235797    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:18.236138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.236102    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.327734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.237839    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:18.327734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.237901    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:18.351671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.351545    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.385673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.379770    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:12:18.385673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.379855    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:12:18.405675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.400386    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:18.405675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.400445    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:18.529876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.529818    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.636928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.636888    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.667185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.666977    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:12:18.687208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.667897    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.744974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.744930    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.760695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.759270    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:12:18.761962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.761916    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.762374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.762336    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:12:18.925729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.916060    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:19.000674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.993777    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.000674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.994184    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.000674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.994462    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.000674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:18.996261    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 15:12:19.229615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.229518    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:19.229920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.229884    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:19.410672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.400036    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.413894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.413864    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:19.414109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.414082    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:19.661797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.661751    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.739478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.663723    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.740076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.740051    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:12:19.740699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.684018    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:19.762597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.762556    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.770618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.770577    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:19.774794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.774345    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:12:20.042723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:19.998304    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.042723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.008503    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.042723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.009872    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.356724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.356262    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:20.356724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.356327    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:20.393682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.358544    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.502700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.495444    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 02 15:12:20.502700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.495510    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 02 15:12:20.698674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.698219    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.716617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.716581    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:20.716931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.716902    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.744173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.744135    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.745663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.745615    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.772765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.770790    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:20.774605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.774577    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:12:20.802558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:20.802516    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.004724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.003752    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.024496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.009721    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.039672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.036945    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.280735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.277838    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:21.280735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.277943    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:21.763465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.726737    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.763465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.727914    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:12:21.763465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.759412    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.814597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.780962    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:21.814597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.781298    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.814597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.807779    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:12:21.814597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:21.808122    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:12:22.010493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.010450    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:12:22.028030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.013055    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:22.039714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.039625    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.211812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.185121    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:22.212277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.212252    7019 patch_prober.go:29] interesting pod/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:22.212510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.212486    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.241722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.234004    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:22.241722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.234059    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:22.241722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.234151    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:22.241722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.234176    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.491381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.468859    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:22.491381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.468925    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.659786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.659737    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": context deadline exceeded"
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.728527    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.760512    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.780822    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.780890    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.781246    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.781597    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.836382    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:22.873892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:22.836733    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:23.037912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.034033    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:23.037912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.037746    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:12:23.055257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.055215    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:12:23.135333 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:12:23.135186    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:12:23.245771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.245735    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:23.246044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.246017    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:23.385242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.385212    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:23.539983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.535785    7019 patch_prober.go:29] interesting pod/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:23.539983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.535865    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:23.540765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.540626    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:23.577506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.576343    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:12:23.580437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.580410    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:12:23.617268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.616789    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:12:23.643858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.643828    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:12:23.649003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.647130    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:12:23.677150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.677124    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:12:23.687464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.687426    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:23.688892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.688873    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:23.695591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.694234    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:12:23.699168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.698543    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:23.718679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.718151    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:12:23.737303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.737277    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:12:23.738570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:23.738550    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:12:24.117568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:24.117284    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:24.230028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:24.229993    7019 patch_prober.go:29] interesting pod/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 02 15:12:24.230297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:24.230268    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 02 15:12:24.317987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:24.317846    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:24.489727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:24.481299    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:25.019671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:25.019129    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:25.243778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:25.242683    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:25.542711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:25.542678    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:25.544233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:25.544200    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:12:25.780338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:25.780292    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:25.952408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:25.952373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:26.297240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:26.297104    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:26.564572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:26.564465    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:27.922657 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:27.922598    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:28.065365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:28.065330    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:28.230896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:28.230811    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:28.839191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:28.839151    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:29.002010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:29.001974    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:29.620141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:29.620110    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:29.782669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:29.781946    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:30.412891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:30.412857    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:30.710732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:30.706104    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:31.583189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:31.583151    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:31.727141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:31.727104    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:32.687239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:32.687195    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:33.874669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:33.874281    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:34.022845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:34.022806    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:34.193506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:34.193424    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:34.391744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:34.391710    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:34.818706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:34.815067    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:34.991670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:34.990070    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:36.209738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:36.209704    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:36.489527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:36.489439    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:36.949151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:36.949113    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"]
Nov 02 15:12:36.954897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:36.954872    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:36.975851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:36.975740    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d3bd1bf1-openshift-default-dockercfg-g9klq\""
Nov 02 15:12:36.997775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:36.997746    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"]
Nov 02 15:12:37.062614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062577    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-certs\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.062847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062633    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nhkl\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-kube-api-access-8nhkl\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.062847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062728    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b96d0d4-9312-4418-b4b8-486709be7a79-istiod-ca-cert\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.062847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062832    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-socket\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.062977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062859    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:37.062977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062868    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-credential-socket\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.062977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062950    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-envoy\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.063084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.062978    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-token\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.063084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.063006    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-podinfo\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.063158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.063094    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-data\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.164724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.164692    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-podinfo\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.164922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.164758    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-data\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.164922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.164840    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-certs\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.164922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.164864    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nhkl\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-kube-api-access-8nhkl\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.164937    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b96d0d4-9312-4418-b4b8-486709be7a79-istiod-ca-cert\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165029    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-socket\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165081    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-credential-socket\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165124    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-envoy\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165149    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-data\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165190    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-certs\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165300    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-token\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165350    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-socket\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165489    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-credential-socket\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.165730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.165705    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b96d0d4-9312-4418-b4b8-486709be7a79-istiod-ca-cert\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.167384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.167360    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-envoy\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.167606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.167585    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-podinfo\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.190832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.190801    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nhkl\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-kube-api-access-8nhkl\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.191964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.191910    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-token\") pod \"e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") " pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.242792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.237063    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:37.277312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.277276    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:37.738616 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:12:37.738577    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b96d0d4_9312_4418_b4b8_486709be7a79.slice/crio-89e101528871a533a0b51734f3ff9adee1d7faa2cd010ab1ad5fda597bbba2df WatchSource:0}: Error finding container 89e101528871a533a0b51734f3ff9adee1d7faa2cd010ab1ad5fda597bbba2df: Status 404 returned error can't find the container with id 89e101528871a533a0b51734f3ff9adee1d7faa2cd010ab1ad5fda597bbba2df
Nov 02 15:12:37.742034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.741996    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:37.742134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.742083    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:37.742134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.742125    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:37.742807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.742426    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"]
Nov 02 15:12:37.838642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:37.838605    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:38.016065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:38.015980    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:38.442357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:38.441190    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:38.515993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:38.515942    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw" event={"ID":"1b96d0d4-9312-4418-b4b8-486709be7a79","Type":"ContainerStarted","Data":"55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8"}
Nov 02 15:12:38.516186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:38.516018    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw" event={"ID":"1b96d0d4-9312-4418-b4b8-486709be7a79","Type":"ContainerStarted","Data":"89e101528871a533a0b51734f3ff9adee1d7faa2cd010ab1ad5fda597bbba2df"}
Nov 02 15:12:38.573849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:38.573784    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw" podStartSLOduration=2.573761414 podStartE2EDuration="2.573761414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:38.565943197 +0000 UTC m=+952.819896832" watchObservedRunningTime="2026-11-02 15:12:38.573761414 +0000 UTC m=+952.827715039"
Nov 02 15:12:39.278050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.278012    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:39.284529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.284499    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:39.330881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.330821    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:39.498477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.498447    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"]
Nov 02 15:12:39.502978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.502957    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.524588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.524520    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:39.525572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.525550    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:12:39.534196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.534134    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"]
Nov 02 15:12:39.603544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.603491    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:39.635611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.635556    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed657d0-b5a5-4095-bc45-9d6db59c762a-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.635914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.635893    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.635987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.635937    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.636046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.636026    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlv87\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-kube-api-access-rlv87\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.636101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.636077    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.636167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.636149    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.636241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.636215    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.636339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.636322    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.636418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.636373    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.737969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.737939    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.738135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.738033    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.738135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.738086    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.738229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.738189    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.738289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.738237    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.738987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.738954    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-credential-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.738987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.738977    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-socket\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.739939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.739916    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed657d0-b5a5-4095-bc45-9d6db59c762a-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.741718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.741692    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-podinfo\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.743664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.739099    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed657d0-b5a5-4095-bc45-9d6db59c762a-istiod-ca-cert\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.743907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.743887    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.743989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.743957    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.744052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.744010    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlv87\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-kube-api-access-rlv87\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.745214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.744834    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-certs\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.745214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.745146    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-data\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.752468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.752443    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-envoy\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.771665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.769486    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-token\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.771665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.770107    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlv87\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-kube-api-access-rlv87\") pod \"e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") " pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:39.771665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.770306    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:39.820147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:39.820057    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:40.012814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.000855    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:40.372189 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:12:40.372161    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed657d0_b5a5_4095_bc45_9d6db59c762a.slice/crio-26c9a2da82d246d4b83633fdb7ec3493caab97752c172b85149359b7f46720d4 WatchSource:0}: Error finding container 26c9a2da82d246d4b83633fdb7ec3493caab97752c172b85149359b7f46720d4: Status 404 returned error can't find the container with id 26c9a2da82d246d4b83633fdb7ec3493caab97752c172b85149359b7f46720d4
Nov 02 15:12:40.372949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.372923    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"]
Nov 02 15:12:40.374809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.374777    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:40.374884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.374842    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:40.374884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.374872    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:40.535011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.534974    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" event={"ID":"aed657d0-b5a5-4095-bc45-9d6db59c762a","Type":"ContainerStarted","Data":"7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd"}
Nov 02 15:12:40.535415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.535037    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" event={"ID":"aed657d0-b5a5-4095-bc45-9d6db59c762a","Type":"ContainerStarted","Data":"26c9a2da82d246d4b83633fdb7ec3493caab97752c172b85149359b7f46720d4"}
Nov 02 15:12:40.589394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.589336    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podStartSLOduration=1.589321844 podStartE2EDuration="1.589321844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:40.585782801 +0000 UTC m=+954.839736425" watchObservedRunningTime="2026-11-02 15:12:40.589321844 +0000 UTC m=+954.843275468"
Nov 02 15:12:40.698180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.698089    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:40.820793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.820764    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:40.914150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:40.914113    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:41.229377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:41.229345    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:41.822301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:41.822265    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:41.822758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:41.822327    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 02 15:12:42.597728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:42.597693    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:42.755423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:42.755389    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:42.821165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:42.821125    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:42.821563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:42.821530    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 02 15:12:43.587669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:43.587145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:43.821385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:43.821342    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:43.821550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:43.821394    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 02 15:12:44.763616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:44.763586    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:44.820519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:44.820468    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:44.820737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:44.820548    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 02 15:12:44.931780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:44.931739    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:45.076288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:45.076205    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:45.234835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:45.234806    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:45.509335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:45.509303    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:45.700785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:45.700747    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:45.820557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:45.820485    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:45.821020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:45.820925    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 02 15:12:45.856927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:45.856897    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:46.822429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:46.821757    7019 patch_prober.go:29] interesting pod/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 02 15:12:46.822429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:46.821877    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 02 15:12:46.922088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:46.922056    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:47.506309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.506278    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:47.574598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.574559    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"]
Nov 02 15:12:47.598677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.595124    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.622828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.622790    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.623124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.623099    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.623979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.623955    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.624123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.624107    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005828cd-e88b-4f49-85fe-957713a31b47-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.624264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.624249    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.624447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.624429    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm54j\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-kube-api-access-cm54j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.624717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.624678    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.641874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.637575    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 15:12:47.642366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.637640    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 15:12:47.642366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.642183    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 15:12:47.642366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.642334    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rcl8t\""
Nov 02 15:12:47.676100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.675845    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"]
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.738594    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.738748    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm54j\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-kube-api-access-cm54j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.738897    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.738980    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.739062    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.739187    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.739229    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005828cd-e88b-4f49-85fe-957713a31b47-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.752497    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.754313    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.777234    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.786257    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.819041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.818519    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:47.849154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.847736    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005828cd-e88b-4f49-85fe-957713a31b47-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.849154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.848526    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm54j\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-kube-api-access-cm54j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.853387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.853335    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-dkghl\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:47.983668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:47.982889    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:48.132368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:48.132332    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:48.133923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:48.133905    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:48.135106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:48.135089    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:12:48.342944 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:12:48.342909    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005828cd_e88b_4f49_85fe_957713a31b47.slice/crio-790014dbb571f3781b8f41508e59996d41f0552c11e825786a2f5f9f785c055b WatchSource:0}: Error finding container 790014dbb571f3781b8f41508e59996d41f0552c11e825786a2f5f9f785c055b: Status 404 returned error can't find the container with id 790014dbb571f3781b8f41508e59996d41f0552c11e825786a2f5f9f785c055b
Nov 02 15:12:48.343871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:48.343844    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"]
Nov 02 15:12:48.657316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:48.657279    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl" event={"ID":"005828cd-e88b-4f49-85fe-957713a31b47","Type":"ContainerStarted","Data":"790014dbb571f3781b8f41508e59996d41f0552c11e825786a2f5f9f785c055b"}
Nov 02 15:12:49.683920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:49.683884    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:51.355825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:51.355747    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:51.982522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:51.982308    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:52.152397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:52.152360    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:52.355712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:52.352204    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:52.520472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:52.520377    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:52.773251 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:12:52.773176    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:12:53.616882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:53.607230    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:53.616882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:53.607316    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:12:53.800682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:53.780137    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:54.445258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:54.444173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:54.638430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:54.638391    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:54.861740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:54.861694    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl" event={"ID":"005828cd-e88b-4f49-85fe-957713a31b47","Type":"ContainerStarted","Data":"27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482"}
Nov 02 15:12:54.861930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:54.861758    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:54.946732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:54.946617    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl" podStartSLOduration=2.6863403999999997 podStartE2EDuration="7.94660191s" totalImagesPullingTime="5.26026151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:54.92974489 +0000 UTC m=+969.183698514" watchObservedRunningTime="2026-11-02 15:12:54.94660191 +0000 UTC m=+969.200555534"
Nov 02 15:12:55.880877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:55.880847    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:12:56.147332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:56.147240    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:57.912201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:57.912163    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:58.670328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:58.670289    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:58.829193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:58.829153    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:59.342296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:59.342263    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:59.494578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:59.494530    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:12:59.633385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:12:59.633345    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:00.131425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:00.131388    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:01.400471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:01.400421    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:03.168110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:03.168079    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:03.333229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:03.333193    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:04.998673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:04.998170    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:06.622804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:06.622770    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:08.374929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:08.374894    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:10.070843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:10.070809    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:11.955845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:11.955807    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:13.707716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.707681    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"]
Nov 02 15:13:13.712239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.712220    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.731777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.731752    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"]
Nov 02 15:13:13.740733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.740708    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:13:13.740733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.740709    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:13:13.747464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.747434    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59ng\" (UniqueName: \"kubernetes.io/projected/61e5435b-282f-464d-b728-e37b761ba8af-kube-api-access-n59ng\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.748111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.748073    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.748305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.748277    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.837459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.837420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:13.860296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.860259    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.860492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.860316    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.860598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.860584    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59ng\" (UniqueName: \"kubernetes.io/projected/61e5435b-282f-464d-b728-e37b761ba8af-kube-api-access-n59ng\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.861639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.861527    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.869388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.869351    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:13.886245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:13.886216    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59ng\" (UniqueName: \"kubernetes.io/projected/61e5435b-282f-464d-b728-e37b761ba8af-kube-api-access-n59ng\") pod \"maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:14.025607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:14.025519    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:14.330836 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:13:14.330804    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e5435b_282f_464d_b728_e37b761ba8af.slice/crio-0651e12c315d77959e6aa14c3dff6ef66d5f80b01fc810c518c4c9ac807863b8 WatchSource:0}: Error finding container 0651e12c315d77959e6aa14c3dff6ef66d5f80b01fc810c518c4c9ac807863b8: Status 404 returned error can't find the container with id 0651e12c315d77959e6aa14c3dff6ef66d5f80b01fc810c518c4c9ac807863b8
Nov 02 15:13:14.331463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:14.331433    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"]
Nov 02 15:13:15.044634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:15.044602    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4" event={"ID":"61e5435b-282f-464d-b728-e37b761ba8af","Type":"ContainerStarted","Data":"baaf9ed0060bfaf46aead267517b20e54ee4aa2bf47b15c790457b2e23fc47a9"}
Nov 02 15:13:15.045075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:15.044676    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4" event={"ID":"61e5435b-282f-464d-b728-e37b761ba8af","Type":"ContainerStarted","Data":"0651e12c315d77959e6aa14c3dff6ef66d5f80b01fc810c518c4c9ac807863b8"}
Nov 02 15:13:15.093519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:15.093470    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4" podStartSLOduration=2.093455569 podStartE2EDuration="2.093455569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:15.089959763 +0000 UTC m=+989.343913387" watchObservedRunningTime="2026-11-02 15:13:15.093455569 +0000 UTC m=+989.347409194"
Nov 02 15:13:15.572525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:15.572481    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:16.053863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:16.053828    7019 generic.go:356] "Generic (PLEG): container finished" podID="61e5435b-282f-464d-b728-e37b761ba8af" containerID="baaf9ed0060bfaf46aead267517b20e54ee4aa2bf47b15c790457b2e23fc47a9" exitCode=0
Nov 02 15:13:16.054345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:16.053886    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4" event={"ID":"61e5435b-282f-464d-b728-e37b761ba8af","Type":"ContainerDied","Data":"baaf9ed0060bfaf46aead267517b20e54ee4aa2bf47b15c790457b2e23fc47a9"}
Nov 02 15:13:17.181121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.181085    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:17.289250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.289223    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:17.437564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.437534    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-openshift-ca\") pod \"61e5435b-282f-464d-b728-e37b761ba8af\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") "
Nov 02 15:13:17.437766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.437708    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-xks-ca\") pod \"61e5435b-282f-464d-b728-e37b761ba8af\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") "
Nov 02 15:13:17.437766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.437754    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e5435b-282f-464d-b728-e37b761ba8af-kube-api-access-n59ng\" (UniqueName: \"kubernetes.io/projected/61e5435b-282f-464d-b728-e37b761ba8af-kube-api-access-n59ng\") pod \"61e5435b-282f-464d-b728-e37b761ba8af\" (UID: \"61e5435b-282f-464d-b728-e37b761ba8af\") "
Nov 02 15:13:17.437961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.437929    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-openshift-ca" pod "61e5435b-282f-464d-b728-e37b761ba8af" (UID: "61e5435b-282f-464d-b728-e37b761ba8af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:17.438319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.438298    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:17.439943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.439908    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-xks-ca" pod "61e5435b-282f-464d-b728-e37b761ba8af" (UID: "61e5435b-282f-464d-b728-e37b761ba8af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:17.440076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.440043    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e5435b-282f-464d-b728-e37b761ba8af-kube-api-access-n59ng" pod "61e5435b-282f-464d-b728-e37b761ba8af" (UID: "61e5435b-282f-464d-b728-e37b761ba8af"). InnerVolumeSpecName "kube-api-access-n59ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:17.540086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.540049    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59ng\" (UniqueName: \"kubernetes.io/projected/61e5435b-282f-464d-b728-e37b761ba8af-kube-api-access-n59ng\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:17.540086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:17.540086    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61e5435b-282f-464d-b728-e37b761ba8af-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:18.070693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:18.070629    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"
Nov 02 15:13:18.071106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:18.070698    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4" event={"ID":"61e5435b-282f-464d-b728-e37b761ba8af","Type":"ContainerDied","Data":"0651e12c315d77959e6aa14c3dff6ef66d5f80b01fc810c518c4c9ac807863b8"}
Nov 02 15:13:18.071106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:18.070759    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0651e12c315d77959e6aa14c3dff6ef66d5f80b01fc810c518c4c9ac807863b8"
Nov 02 15:13:18.802871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:18.802837    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:20.403082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:20.403045    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:22.178309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:22.178272    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:22.709935 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:13:22.709891    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:13:23.917048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:23.917004    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:24.366850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.366817    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"]
Nov 02 15:13:24.367545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.367529    7019 cpu_manager.go:414] "containerMap: removing container" podUID="61e5435b-282f-464d-b728-e37b761ba8af" containerName="revoke-keys"
Nov 02 15:13:24.367545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.367545    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e5435b-282f-464d-b728-e37b761ba8af" containerName="revoke-keys"
Nov 02 15:13:24.367708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.367693    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e5435b-282f-464d-b728-e37b761ba8af" containerName="revoke-keys"
Nov 02 15:13:24.376466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.376431    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:24.377702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.377685    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.399109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.392878    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"]
Nov 02 15:13:24.401454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.401428    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:13:24.401708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.401689    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:13:24.497637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.497581    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.498936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.498902    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx45q\" (UniqueName: \"kubernetes.io/projected/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-kube-api-access-xx45q\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.499134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.498954    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.527961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.527918    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:24.600614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.600574    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.600835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.600794    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx45q\" (UniqueName: \"kubernetes.io/projected/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-kube-api-access-xx45q\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.600835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.600827    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.601481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.601452    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.603281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.603258    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.626014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.625947    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx45q\" (UniqueName: \"kubernetes.io/projected/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-kube-api-access-xx45q\") pod \"maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:24.702938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:24.702900    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:25.229042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:25.228446    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"]
Nov 02 15:13:25.257680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:25.257621    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:25.409340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:25.409306    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:26.141812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:26.141774    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb" event={"ID":"f7590c43-fbb1-42e6-bfbf-be45c8c02b26","Type":"ContainerStarted","Data":"f592f4868dfd2df48f355ea0bf8a1dea1f26de01c88c44da26844eba043059c2"}
Nov 02 15:13:26.141995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:26.141830    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb" event={"ID":"f7590c43-fbb1-42e6-bfbf-be45c8c02b26","Type":"ContainerStarted","Data":"d821294754ffd6309295f7d3d499af9e9185890f6b243763016ed499deaad0f2"}
Nov 02 15:13:26.189236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:26.189139    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb" podStartSLOduration=2.189120011 podStartE2EDuration="2.189120011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:26.18442632 +0000 UTC m=+1000.438379968" watchObservedRunningTime="2026-11-02 15:13:26.189120011 +0000 UTC m=+1000.443073652"
Nov 02 15:13:26.260261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:26.260227    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:26.967123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:26.967037    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:13:27.198670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:27.193445    7019 generic.go:356] "Generic (PLEG): container finished" podID="f7590c43-fbb1-42e6-bfbf-be45c8c02b26" containerID="f592f4868dfd2df48f355ea0bf8a1dea1f26de01c88c44da26844eba043059c2" exitCode=0
Nov 02 15:13:27.198670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:27.193594    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb" event={"ID":"f7590c43-fbb1-42e6-bfbf-be45c8c02b26","Type":"ContainerDied","Data":"f592f4868dfd2df48f355ea0bf8a1dea1f26de01c88c44da26844eba043059c2"}
Nov 02 15:13:29.126671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.102301    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:29.523734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.522860    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:29.672677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.667849    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-openshift-ca\") pod \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") "
Nov 02 15:13:29.672677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.667974    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-xks-ca\") pod \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") "
Nov 02 15:13:29.672677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.668180    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-kube-api-access-xx45q\" (UniqueName: \"kubernetes.io/projected/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-kube-api-access-xx45q\") pod \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\" (UID: \"f7590c43-fbb1-42e6-bfbf-be45c8c02b26\") "
Nov 02 15:13:29.718523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.718474    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-openshift-ca" pod "f7590c43-fbb1-42e6-bfbf-be45c8c02b26" (UID: "f7590c43-fbb1-42e6-bfbf-be45c8c02b26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:29.719505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.719479    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:29.720756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.720726    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Nov 02 15:13:29.763833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.762361    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-kube-api-access-xx45q" pod "f7590c43-fbb1-42e6-bfbf-be45c8c02b26" (UID: "f7590c43-fbb1-42e6-bfbf-be45c8c02b26"). InnerVolumeSpecName "kube-api-access-xx45q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:29.785961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.785842    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:13:29.833159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.806094    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:13:29.833159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.806313    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:13:29.845008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.838205    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx45q\" (UniqueName: \"kubernetes.io/projected/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-kube-api-access-xx45q\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:29.864550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.849532    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:13:29.865962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.865914    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:29.876318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.876281    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-xks-ca" pod "f7590c43-fbb1-42e6-bfbf-be45c8c02b26" (UID: "f7590c43-fbb1-42e6-bfbf-be45c8c02b26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:13:29.972662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:29.972607    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7590c43-fbb1-42e6-bfbf-be45c8c02b26-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:30.016478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:30.015852    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:30.035759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:30.035085    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:13:30.274671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:30.253874    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:13:30.429606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:30.429481    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb" event={"ID":"f7590c43-fbb1-42e6-bfbf-be45c8c02b26","Type":"ContainerDied","Data":"d821294754ffd6309295f7d3d499af9e9185890f6b243763016ed499deaad0f2"}
Nov 02 15:13:30.430196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:30.430123    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"
Nov 02 15:13:30.444479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:30.438800    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d821294754ffd6309295f7d3d499af9e9185890f6b243763016ed499deaad0f2"
Nov 02 15:13:30.885668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:30.885003    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:31.284730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:31.280863    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:31.596730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:31.585067    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:31.763005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:31.762923    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:31.788907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:31.788875    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"]
Nov 02 15:13:31.869893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:31.869853    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:31.998674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:31.997913    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:32.003375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.003344    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0621b336-tt7z4"]
Nov 02 15:13:32.026691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.017248    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.063803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.063762    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.496674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.491400    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:32.587071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.587030    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.700145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.700106    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:32.763837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.763733    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.765965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.764080    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.801489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.801445    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:32.834807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.834770    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:13:32.835134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:32.835102    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.272708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.269894    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:33.455032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.447614    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:33.634686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.623305    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:33.769851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.769793    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.799566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.799527    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:33.875683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.812309    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:13:33.875683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.812382    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:13:33.875683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.814196    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:13:33.875683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.826553    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:13:34.007753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:33.980730    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e5435b-282f-464d-b728-e37b761ba8af" path="/var/lib/kubelet/pods/61e5435b-282f-464d-b728-e37b761ba8af/volumes"
Nov 02 15:13:34.061737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.019387    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:34.062046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.023600    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.064448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.054093    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:34.336729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.335057    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:13:34.431292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.430228    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:34.646572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.646540    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:34.733670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.728397    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:13:34.771353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.771309    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.817694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.813102    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 02 15:13:34.828795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:34.828468    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:35.732671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:35.731814    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:35.787059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:35.787019    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:13:35.826634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:35.826604    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:13:36.056375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:36.056277    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:37.584729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:37.584691    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:39.340439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:39.340400    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:40.707181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:40.707144    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:40.854757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:40.854720    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:41.225926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:41.225887    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:42.512243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:42.512202    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:42.895948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:42.895917    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:43.666510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:43.666463    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:44.262964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:44.262928    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:44.404776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:44.404742    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:45.231500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:45.231462    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:45.383583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:45.383470    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:46.141749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:46.141711    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:46.206482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:46.206449    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"]
Nov 02 15:13:46.219530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:46.219494    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-649ad0be-7f6sb"]
Nov 02 15:13:46.330728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:46.330017    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:46.518896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:46.512182    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:47.697139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:47.697100    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:47.879518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:47.867412    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7590c43-fbb1-42e6-bfbf-be45c8c02b26" path="/var/lib/kubelet/pods/f7590c43-fbb1-42e6-bfbf-be45c8c02b26/volumes"
Nov 02 15:13:48.032629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:48.032526    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:48.285503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:48.285412    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:48.487715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:48.479218    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:48.661676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:48.652970    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:48.798316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:48.794426    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:50.414897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:50.385469    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"]
Nov 02 15:13:50.415877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:50.415835    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw" podUID="1b96d0d4-9312-4418-b4b8-486709be7a79" containerName="istio-proxy" containerID="cri-o://55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8" gracePeriod=30
Nov 02 15:13:50.429751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:50.429688    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:50.961418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:50.961279    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:51.391185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:51.391154    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:51.553299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:51.553263    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:51.700375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:51.700344    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:52.210939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:52.210873    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:52.419867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:52.419834    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:52.579278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:52.579180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:52.711156 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:13:52.711114    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:13:52.934603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:52.934563    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:54.109425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:54.109386    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:54.275441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:54.275393    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:54.870882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:54.870844    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:55.505206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:55.503045    7019 patch_prober.go:29] interesting pod/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:51858->10.134.0.78:15021: read: connection reset by peer" start-of-body=
Nov 02 15:13:55.505206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:55.503125    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw" podUID="1b96d0d4-9312-4418-b4b8-486709be7a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:51858->10.134.0.78:15021: read: connection reset by peer"
Nov 02 15:13:56.242262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.242227    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:13:56.277553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.277516    7019 generic.go:356] "Generic (PLEG): container finished" podID="1b96d0d4-9312-4418-b4b8-486709be7a79" containerID="55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8" exitCode=0
Nov 02 15:13:56.277553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.277577    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"
Nov 02 15:13:56.277888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.277615    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw" event={"ID":"1b96d0d4-9312-4418-b4b8-486709be7a79","Type":"ContainerDied","Data":"55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8"}
Nov 02 15:13:56.277888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.277697    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw" event={"ID":"1b96d0d4-9312-4418-b4b8-486709be7a79","Type":"ContainerDied","Data":"89e101528871a533a0b51734f3ff9adee1d7faa2cd010ab1ad5fda597bbba2df"}
Nov 02 15:13:56.277888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.277749    7019 scope.go:122] "RemoveContainer" containerID="55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8"
Nov 02 15:13:56.301663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.301622    7019 scope.go:122] "RemoveContainer" containerID="55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8"
Nov 02 15:13:56.304042 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:13:56.302211    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8\": container with ID starting with 55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8 not found: ID does not exist" containerID="55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8"
Nov 02 15:13:56.304278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.304244    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8"} err="failed to get container status \"55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8\": rpc error: code = NotFound desc = could not find container \"55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8\": container with ID starting with 55c0f26dcf953691131e8a0836c6a301d49c6314837a4ac3b83430bb54d07ae8 not found: ID does not exist"
Nov 02 15:13:56.350780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.350746    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-credential-socket\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.350987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.350871    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-token\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-token\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.351070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351033    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-podinfo\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.351129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351065    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-credential-socket" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:56.351186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351145    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-certs\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.351242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351195    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-socket\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.351303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351250    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-data\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.351357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351302    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b96d0d4-9312-4418-b4b8-486709be7a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b96d0d4-9312-4418-b4b8-486709be7a79-istiod-ca-cert\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.351441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351420    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-kube-api-access-8nhkl\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-kube-api-access-8nhkl\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.351498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351460    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-envoy\") pod \"1b96d0d4-9312-4418-b4b8-486709be7a79\" (UID: \"1b96d0d4-9312-4418-b4b8-486709be7a79\") "
Nov 02 15:13:56.352266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.352238    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.352388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351459    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-socket" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:56.352388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351481    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-certs" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:56.352388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351776    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b96d0d4-9312-4418-b4b8-486709be7a79-istiod-ca-cert" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:13:56.352388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.351982    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-data" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:56.354339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.354312    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-kube-api-access-8nhkl" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "kube-api-access-8nhkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:56.354590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.354561    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-token" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:13:56.354812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.354792    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-podinfo" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:13:56.355056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.355032    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-envoy" pod "1b96d0d4-9312-4418-b4b8-486709be7a79" (UID: "1b96d0d4-9312-4418-b4b8-486709be7a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:13:56.453636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453598    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.453636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453639    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.453857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453663    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b96d0d4-9312-4418-b4b8-486709be7a79-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.453857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453674    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nhkl\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-kube-api-access-8nhkl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.453857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453684    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.453857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453693    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.453857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453700    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b96d0d4-9312-4418-b4b8-486709be7a79-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.453857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.453708    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b96d0d4-9312-4418-b4b8-486709be7a79-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:13:56.631272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.631238    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"]
Nov 02 15:13:56.644809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.644781    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d3bd1bf1-openshift-default-69587fc7f9-lgxnw"]
Nov 02 15:13:56.685171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:56.685133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:57.864397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:57.864364    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b96d0d4-9312-4418-b4b8-486709be7a79" path="/var/lib/kubelet/pods/1b96d0d4-9312-4418-b4b8-486709be7a79/volumes"
Nov 02 15:13:58.345676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:58.345573    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:58.484494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:58.484458    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:58.625457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:58.625424    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:13:59.481922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:13:59.479829    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:00.106818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:00.106782    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:01.747033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:01.746993    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:03.369460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:03.369429    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:04.907957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:04.907911    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:06.815236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:06.815201    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:08.856511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:08.856477    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:09.002526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:09.002494    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:10.780963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:10.780931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:12.486801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:12.486743    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:14.083472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:14.083437    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:15.755890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:15.755853    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:17.099538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:17.099495    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:17.272355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:17.272317    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:17.565277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:17.565242    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:18.479156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:18.479120    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:19.293794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:19.293748    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:20.995356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:20.995318    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:22.566331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:22.566296    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:22.725093 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:14:22.725052    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:14:24.164749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:24.164714    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:25.561352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:25.561313    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:25.831818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:25.831731    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:27.542509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:27.542461    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:28.030417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:28.030380    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:28.175062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:28.175016    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:28.308472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:28.308385    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:29.300412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:29.300355    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:30.993301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:30.993258    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:32.695888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:32.695852    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:32.967321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:32.967207    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:33.320733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:33.320241    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:14:35.132342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:35.132304    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:35.282770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:35.282733    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:36.872090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:36.872057    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:38.691842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:38.691809    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:40.490087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:40.490050    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:41.937431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:41.937393    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:42.347140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:42.347052    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:42.869417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:42.869382    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:43.016075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:43.016043    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:43.454916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:43.454881    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:43.934885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:43.934850    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:44.288907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:44.288824    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:44.697481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:44.697446    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:45.337826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:45.337787    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:46.391968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:46.391931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:48.306113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:48.306073    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:48.509389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:48.509357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:50.019709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:50.019666    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:50.292824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:50.292739    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:50.697865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:50.697807    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:52.354417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:52.354373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:52.734087 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:14:52.733833    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:14:53.143412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:53.143380    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:53.315473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:53.315439    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:53.455078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:53.455008    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:54.024512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:54.024480    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:54.841435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:54.841400    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:55.408165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:55.408128    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:55.584458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:55.584422    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:55.771195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:55.771108    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:57.290977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:57.290886    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:59.451921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:59.451879    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:59.640869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:59.640833    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:14:59.804727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:14:59.797453    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:00.210518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.208684    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"]
Nov 02 15:15:00.211175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.211106    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f7590c43-fbb1-42e6-bfbf-be45c8c02b26" containerName="revoke-keys"
Nov 02 15:15:00.211175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.211131    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7590c43-fbb1-42e6-bfbf-be45c8c02b26" containerName="revoke-keys"
Nov 02 15:15:00.212673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.211429    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1b96d0d4-9312-4418-b4b8-486709be7a79" containerName="istio-proxy"
Nov 02 15:15:00.212673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.211446    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b96d0d4-9312-4418-b4b8-486709be7a79" containerName="istio-proxy"
Nov 02 15:15:00.212673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.211672    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b96d0d4-9312-4418-b4b8-486709be7a79" containerName="istio-proxy"
Nov 02 15:15:00.212673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.211702    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7590c43-fbb1-42e6-bfbf-be45c8c02b26" containerName="revoke-keys"
Nov 02 15:15:00.320674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.216223    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.320674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.254760    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"]
Nov 02 15:15:00.320674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.319439    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:15:00.340610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.340582    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"]
Nov 02 15:15:00.340943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.340923    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:00.364147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.364114    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0569616b-19d3-49b0-ac1d-58fc40dd61d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.364524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.364496    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46whx\" (UniqueName: \"kubernetes.io/projected/0569616b-19d3-49b0-ac1d-58fc40dd61d8-kube-api-access-46whx\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.543725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.514240    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a2c650-c22a-4b4e-a736-5fc1e23e136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:00.543725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.514446    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvx4z\" (UniqueName: \"kubernetes.io/projected/52a2c650-c22a-4b4e-a736-5fc1e23e136f-kube-api-access-bvx4z\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:00.543725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.514808    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0569616b-19d3-49b0-ac1d-58fc40dd61d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.543725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.515075    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46whx\" (UniqueName: \"kubernetes.io/projected/0569616b-19d3-49b0-ac1d-58fc40dd61d8-kube-api-access-46whx\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.543725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.540060    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0569616b-19d3-49b0-ac1d-58fc40dd61d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.547187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.547158    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"]
Nov 02 15:15:00.571807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.571771    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46whx\" (UniqueName: \"kubernetes.io/projected/0569616b-19d3-49b0-ac1d-58fc40dd61d8-kube-api-access-46whx\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.638926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.638866    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a2c650-c22a-4b4e-a736-5fc1e23e136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:00.638926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.638924    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvx4z\" (UniqueName: \"kubernetes.io/projected/52a2c650-c22a-4b4e-a736-5fc1e23e136f-kube-api-access-bvx4z\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:00.640326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.640298    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a2c650-c22a-4b4e-a736-5fc1e23e136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:00.659484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.659450    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:00.708569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.708537    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvx4z\" (UniqueName: \"kubernetes.io/projected/52a2c650-c22a-4b4e-a736-5fc1e23e136f-kube-api-access-bvx4z\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:00.750385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:00.750349    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:01.056341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:01.055951    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"]
Nov 02 15:15:01.056640 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:15:01.056615    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0569616b_19d3_49b0_ac1d_58fc40dd61d8.slice/crio-bbd94f7a4a3bcb60bda9d90b3557d447b7b5fd98a424b60c5bb5c7b813cf0696 WatchSource:0}: Error finding container bbd94f7a4a3bcb60bda9d90b3557d447b7b5fd98a424b60c5bb5c7b813cf0696: Status 404 returned error can't find the container with id bbd94f7a4a3bcb60bda9d90b3557d447b7b5fd98a424b60c5bb5c7b813cf0696
Nov 02 15:15:01.163816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:01.163787    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"]
Nov 02 15:15:01.165392 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:15:01.165345    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a2c650_c22a_4b4e_a736_5fc1e23e136f.slice/crio-9b5fef3ce3f44a97c57c295a09257dcde1c469cb6ad612aa9b1e79279d6888c6 WatchSource:0}: Error finding container 9b5fef3ce3f44a97c57c295a09257dcde1c469cb6ad612aa9b1e79279d6888c6: Status 404 returned error can't find the container with id 9b5fef3ce3f44a97c57c295a09257dcde1c469cb6ad612aa9b1e79279d6888c6
Nov 02 15:15:01.193500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:01.193461    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerStarted","Data":"9b5fef3ce3f44a97c57c295a09257dcde1c469cb6ad612aa9b1e79279d6888c6"}
Nov 02 15:15:01.194884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:01.194857    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerStarted","Data":"bbd94f7a4a3bcb60bda9d90b3557d447b7b5fd98a424b60c5bb5c7b813cf0696"}
Nov 02 15:15:01.499118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:01.499065    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:01.710675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:01.706243    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:02.252880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:02.252847    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerStarted","Data":"a2a95197b3086785cfe4923f747b1c51ee62abdcc0d29d752d5649eda22f8aff"}
Nov 02 15:15:02.254769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:02.254745    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerStarted","Data":"06c1a787e7797e304c50f1a74764a97a999b04a2113fc7c063f59c126204bbfa"}
Nov 02 15:15:02.255101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:02.255087    7019 scope.go:122] "RemoveContainer" containerID="06c1a787e7797e304c50f1a74764a97a999b04a2113fc7c063f59c126204bbfa"
Nov 02 15:15:02.396474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:02.396404    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" podStartSLOduration=2.396385163 podStartE2EDuration="2.396385163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:02.306385805 +0000 UTC m=+1096.560339431" watchObservedRunningTime="2026-11-02 15:15:02.396385163 +0000 UTC m=+1096.650338786"
Nov 02 15:15:02.959009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:02.958971    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:03.267791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:03.267700    7019 generic.go:356] "Generic (PLEG): container finished" podID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerID="06c1a787e7797e304c50f1a74764a97a999b04a2113fc7c063f59c126204bbfa" exitCode=60
Nov 02 15:15:03.267791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:03.267767    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerDied","Data":"06c1a787e7797e304c50f1a74764a97a999b04a2113fc7c063f59c126204bbfa"}
Nov 02 15:15:03.268026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:03.267841    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerStarted","Data":"d532ba3f8bee9ce78d5fdcf39417d786f5a615614d76c1d9cc596597f4a5ef25"}
Nov 02 15:15:03.270500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:03.270445    7019 generic.go:356] "Generic (PLEG): container finished" podID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerID="a2a95197b3086785cfe4923f747b1c51ee62abdcc0d29d752d5649eda22f8aff" exitCode=60
Nov 02 15:15:03.270742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:03.270544    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerDied","Data":"a2a95197b3086785cfe4923f747b1c51ee62abdcc0d29d752d5649eda22f8aff"}
Nov 02 15:15:03.270896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:03.270872    7019 scope.go:122] "RemoveContainer" containerID="a2a95197b3086785cfe4923f747b1c51ee62abdcc0d29d752d5649eda22f8aff"
Nov 02 15:15:03.316868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:03.316803    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" podStartSLOduration=3.316784659 podStartE2EDuration="3.316784659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:03.315925726 +0000 UTC m=+1097.569879351" watchObservedRunningTime="2026-11-02 15:15:03.316784659 +0000 UTC m=+1097.570738285"
Nov 02 15:15:04.297833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:04.297802    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerStarted","Data":"fe0a91561c9e9c8a44c68adbd289e584675f75e60e101c5799a7e1dd1ed268d9"}
Nov 02 15:15:04.300585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:04.300551    7019 generic.go:356] "Generic (PLEG): container finished" podID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerID="d532ba3f8bee9ce78d5fdcf39417d786f5a615614d76c1d9cc596597f4a5ef25" exitCode=60
Nov 02 15:15:04.300759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:04.300620    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerDied","Data":"d532ba3f8bee9ce78d5fdcf39417d786f5a615614d76c1d9cc596597f4a5ef25"}
Nov 02 15:15:04.300759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:04.300698    7019 scope.go:122] "RemoveContainer" containerID="06c1a787e7797e304c50f1a74764a97a999b04a2113fc7c063f59c126204bbfa"
Nov 02 15:15:04.301010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:04.300993    7019 scope.go:122] "RemoveContainer" containerID="d532ba3f8bee9ce78d5fdcf39417d786f5a615614d76c1d9cc596597f4a5ef25"
Nov 02 15:15:04.301383 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:04.301362    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24_odh-ai-gateway-infra(0569616b-19d3-49b0-ac1d-58fc40dd61d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8"
Nov 02 15:15:04.919373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:04.919321    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:05.314620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:05.314524    7019 generic.go:356] "Generic (PLEG): container finished" podID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerID="fe0a91561c9e9c8a44c68adbd289e584675f75e60e101c5799a7e1dd1ed268d9" exitCode=60
Nov 02 15:15:05.315118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:05.314604    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerDied","Data":"fe0a91561c9e9c8a44c68adbd289e584675f75e60e101c5799a7e1dd1ed268d9"}
Nov 02 15:15:05.315118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:05.314700    7019 scope.go:122] "RemoveContainer" containerID="a2a95197b3086785cfe4923f747b1c51ee62abdcc0d29d752d5649eda22f8aff"
Nov 02 15:15:05.315118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:05.315112    7019 scope.go:122] "RemoveContainer" containerID="fe0a91561c9e9c8a44c68adbd289e584675f75e60e101c5799a7e1dd1ed268d9"
Nov 02 15:15:05.315572 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:05.315546    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85_odh-ai-gateway-infra(52a2c650-c22a-4b4e-a736-5fc1e23e136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f"
Nov 02 15:15:05.317261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:05.317245    7019 scope.go:122] "RemoveContainer" containerID="d532ba3f8bee9ce78d5fdcf39417d786f5a615614d76c1d9cc596597f4a5ef25"
Nov 02 15:15:05.317623 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:05.317590    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24_odh-ai-gateway-infra(0569616b-19d3-49b0-ac1d-58fc40dd61d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8"
Nov 02 15:15:06.334008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:06.333970    7019 scope.go:122] "RemoveContainer" containerID="fe0a91561c9e9c8a44c68adbd289e584675f75e60e101c5799a7e1dd1ed268d9"
Nov 02 15:15:06.334523 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:06.334447    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85_odh-ai-gateway-infra(52a2c650-c22a-4b4e-a736-5fc1e23e136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f"
Nov 02 15:15:06.728678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:06.726599    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:08.456733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:08.453352    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:08.619605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:08.619543    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:08.768972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:08.768883    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:10.273642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:10.273600    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:12.021537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:12.021491    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:13.659437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:13.659383    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:14.073717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:14.073609    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:14.854414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:14.854385    7019 scope.go:122] "RemoveContainer" containerID="d532ba3f8bee9ce78d5fdcf39417d786f5a615614d76c1d9cc596597f4a5ef25"
Nov 02 15:15:15.302002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:15.301961    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:15.416861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:15.416826    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerStarted","Data":"7043ce564c6f7491d27351d42dff62f27ac993705df1179457471ce6b392f7e3"}
Nov 02 15:15:15.860481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:15.860452    7019 scope.go:122] "RemoveContainer" containerID="fe0a91561c9e9c8a44c68adbd289e584675f75e60e101c5799a7e1dd1ed268d9"
Nov 02 15:15:16.425843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:16.425805    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerStarted","Data":"6084c7bcfcd3a284042f8e67381c3aaee36a7e6bedfca275d82c03bcda6bc152"}
Nov 02 15:15:16.427802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:16.427773    7019 generic.go:356] "Generic (PLEG): container finished" podID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerID="7043ce564c6f7491d27351d42dff62f27ac993705df1179457471ce6b392f7e3" exitCode=60
Nov 02 15:15:16.427937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:16.427845    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerDied","Data":"7043ce564c6f7491d27351d42dff62f27ac993705df1179457471ce6b392f7e3"}
Nov 02 15:15:16.427937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:16.427899    7019 scope.go:122] "RemoveContainer" containerID="d532ba3f8bee9ce78d5fdcf39417d786f5a615614d76c1d9cc596597f4a5ef25"
Nov 02 15:15:16.428266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:16.428248    7019 scope.go:122] "RemoveContainer" containerID="7043ce564c6f7491d27351d42dff62f27ac993705df1179457471ce6b392f7e3"
Nov 02 15:15:16.428608 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:16.428588    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24_odh-ai-gateway-infra(0569616b-19d3-49b0-ac1d-58fc40dd61d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8"
Nov 02 15:15:16.477092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:16.477061    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"]
Nov 02 15:15:16.902536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:16.902486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:17.439761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.439724    7019 generic.go:356] "Generic (PLEG): container finished" podID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerID="6084c7bcfcd3a284042f8e67381c3aaee36a7e6bedfca275d82c03bcda6bc152" exitCode=60
Nov 02 15:15:17.439960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.439791    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerDied","Data":"6084c7bcfcd3a284042f8e67381c3aaee36a7e6bedfca275d82c03bcda6bc152"}
Nov 02 15:15:17.439960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.439877    7019 scope.go:122] "RemoveContainer" containerID="fe0a91561c9e9c8a44c68adbd289e584675f75e60e101c5799a7e1dd1ed268d9"
Nov 02 15:15:17.440320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.440300    7019 scope.go:122] "RemoveContainer" containerID="6084c7bcfcd3a284042f8e67381c3aaee36a7e6bedfca275d82c03bcda6bc152"
Nov 02 15:15:17.440824 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:17.440800    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85_odh-ai-gateway-infra(52a2c650-c22a-4b4e-a736-5fc1e23e136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f"
Nov 02 15:15:17.484883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.484851    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"]
Nov 02 15:15:17.664172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.664142    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:17.720057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.719973    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0569616b-19d3-49b0-ac1d-58fc40dd61d8-kube-api-access-46whx\" (UniqueName: \"kubernetes.io/projected/0569616b-19d3-49b0-ac1d-58fc40dd61d8-kube-api-access-46whx\") pod \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") "
Nov 02 15:15:17.720367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.720343    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0569616b-19d3-49b0-ac1d-58fc40dd61d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0569616b-19d3-49b0-ac1d-58fc40dd61d8-maas-api-service-ca\") pod \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\" (UID: \"0569616b-19d3-49b0-ac1d-58fc40dd61d8\") "
Nov 02 15:15:17.721369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.721340    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0569616b-19d3-49b0-ac1d-58fc40dd61d8-maas-api-service-ca" pod "0569616b-19d3-49b0-ac1d-58fc40dd61d8" (UID: "0569616b-19d3-49b0-ac1d-58fc40dd61d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:17.738323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.735488    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0569616b-19d3-49b0-ac1d-58fc40dd61d8-kube-api-access-46whx" pod "0569616b-19d3-49b0-ac1d-58fc40dd61d8" (UID: "0569616b-19d3-49b0-ac1d-58fc40dd61d8"). InnerVolumeSpecName "kube-api-access-46whx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:17.823321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.823277    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0569616b-19d3-49b0-ac1d-58fc40dd61d8-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:15:17.823321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:17.823318    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46whx\" (UniqueName: \"kubernetes.io/projected/0569616b-19d3-49b0-ac1d-58fc40dd61d8-kube-api-access-46whx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:15:18.456001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.455971    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24" event={"ID":"0569616b-19d3-49b0-ac1d-58fc40dd61d8","Type":"ContainerDied","Data":"bbd94f7a4a3bcb60bda9d90b3557d447b7b5fd98a424b60c5bb5c7b813cf0696"}
Nov 02 15:15:18.456494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.456039    7019 scope.go:122] "RemoveContainer" containerID="7043ce564c6f7491d27351d42dff62f27ac993705df1179457471ce6b392f7e3"
Nov 02 15:15:18.456494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.456157    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"
Nov 02 15:15:18.545745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.545716    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"]
Nov 02 15:15:18.549130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.549104    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:18.549639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.549619    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893875-vqx24"]
Nov 02 15:15:18.651840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.651816    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:18.692300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.692256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:18.747706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.741916    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a2c650-c22a-4b4e-a736-5fc1e23e136f-kube-api-access-bvx4z\" (UniqueName: \"kubernetes.io/projected/52a2c650-c22a-4b4e-a736-5fc1e23e136f-kube-api-access-bvx4z\") pod \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") "
Nov 02 15:15:18.747706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.743043    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a2c650-c22a-4b4e-a736-5fc1e23e136f-maas-api-service-ca" pod "52a2c650-c22a-4b4e-a736-5fc1e23e136f" (UID: "52a2c650-c22a-4b4e-a736-5fc1e23e136f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:18.747706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.743123    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a2c650-c22a-4b4e-a736-5fc1e23e136f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a2c650-c22a-4b4e-a736-5fc1e23e136f-maas-api-service-ca\") pod \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\" (UID: \"52a2c650-c22a-4b4e-a736-5fc1e23e136f\") "
Nov 02 15:15:18.747706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.744566    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52a2c650-c22a-4b4e-a736-5fc1e23e136f-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:15:18.762512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.762462    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a2c650-c22a-4b4e-a736-5fc1e23e136f-kube-api-access-bvx4z" pod "52a2c650-c22a-4b4e-a736-5fc1e23e136f" (UID: "52a2c650-c22a-4b4e-a736-5fc1e23e136f"). InnerVolumeSpecName "kube-api-access-bvx4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:18.846080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:18.845966    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvx4z\" (UniqueName: \"kubernetes.io/projected/52a2c650-c22a-4b4e-a736-5fc1e23e136f-kube-api-access-bvx4z\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:15:19.473723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:19.473678    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85" event={"ID":"52a2c650-c22a-4b4e-a736-5fc1e23e136f","Type":"ContainerDied","Data":"9b5fef3ce3f44a97c57c295a09257dcde1c469cb6ad612aa9b1e79279d6888c6"}
Nov 02 15:15:19.474689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:19.474663    7019 scope.go:122] "RemoveContainer" containerID="6084c7bcfcd3a284042f8e67381c3aaee36a7e6bedfca275d82c03bcda6bc152"
Nov 02 15:15:19.474999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:19.474354    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"
Nov 02 15:15:19.543851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:19.543539    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"]
Nov 02 15:15:19.556475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:19.556444    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893875-nmg85"]
Nov 02 15:15:19.872267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:19.872233    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" path="/var/lib/kubelet/pods/0569616b-19d3-49b0-ac1d-58fc40dd61d8/volumes"
Nov 02 15:15:19.872906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:19.872885    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" path="/var/lib/kubelet/pods/52a2c650-c22a-4b4e-a736-5fc1e23e136f/volumes"
Nov 02 15:15:20.656527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:20.656474    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:22.381732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:22.381614    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:22.708071 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:22.707978    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:15:24.245056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:24.245023    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:25.650505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:25.650468    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:27.192969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:27.192918    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:28.920829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:28.920786    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:30.722987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:30.722953    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:32.383051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:32.383018    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:33.982881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:33.982839    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:15:34.072547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:34.072512    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:34.231671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:34.228073    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:36.171843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:36.171805    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:38.210462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:38.210422    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:40.570694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:40.569808    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:42.374453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:42.374380    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:43.996095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:43.996058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:44.140120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:44.140068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:45.014943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.014907    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:45.470097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.470061    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:45.612842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.612796    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:45.885197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.885166    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"]
Nov 02 15:15:45.886123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886097    7019 cpu_manager.go:414] "containerMap: removing container" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886122    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886146    7019 cpu_manager.go:414] "containerMap: removing container" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886154    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886201    7019 cpu_manager.go:414] "containerMap: removing container" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886208    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886223    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886230    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886274    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:15:45.886310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886282    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:15:45.886709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886431    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:15:45.886709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886445    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886456    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:15:45.886709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.886475    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:15:45.937214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.937174    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:45.939477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.939447    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:45.942785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.942759    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"]
Nov 02 15:15:45.961037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:45.961013    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-cce791-openshift-default-dockercfg-w7mhr\""
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.076816    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.076966    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nx98\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-kube-api-access-4nx98\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.077019    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.077059    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.077098    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.077152    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.077198    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.077253    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.086669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.077294    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.134230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.134195    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:46.182785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.182683    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.183196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.183162    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nx98\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-kube-api-access-4nx98\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.183418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.183393    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.183595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.183577    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.183749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.183729    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.183890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.183876    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.184022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.184005    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.184156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.184142    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.184275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.184262    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.201669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.185024    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.201928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.187165    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-data\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.202042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.187371    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-credential-socket\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.202140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.187606    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-certs\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.202228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.188639    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istiod-ca-cert\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.262871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.262793    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-envoy\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.264323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.264259    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nx98\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-kube-api-access-4nx98\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.270676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.270158    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-token\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.270833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.270741    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-podinfo\") pod \"e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") " pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.297224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.291911    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:46.557007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.556923    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:46.615524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.615484    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:46.829316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.828529    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:46.977439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:46.977339    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:47.135518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:47.135476    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"]
Nov 02 15:15:47.140756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:47.140711    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:15:47.140856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:47.140790    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:15:47.140856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:47.140819    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:15:47.720498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:47.720457    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:47.944032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:47.943994    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" event={"ID":"93149e3b-33b5-4cce-bbec-972ffd4b1c61","Type":"ContainerStarted","Data":"1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda"}
Nov 02 15:15:47.944215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:47.944055    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" event={"ID":"93149e3b-33b5-4cce-bbec-972ffd4b1c61","Type":"ContainerStarted","Data":"629ac0a57ef477dfd7e5f14497fd8bf0600357f755c58fc696f1c1525709d81f"}
Nov 02 15:15:48.141642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:48.141541    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:48.558103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:48.557992    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:48.564972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:48.564940    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:48.642665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:48.642594    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podStartSLOduration=3.642575381 podStartE2EDuration="3.642575381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:48.016667466 +0000 UTC m=+1142.270621084" watchObservedRunningTime="2026-11-02 15:15:48.642575381 +0000 UTC m=+1142.896529007"
Nov 02 15:15:48.962505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:48.962462    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:48.963447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:48.963426    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:15:49.924411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:49.924371    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:50.512414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:50.512373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:50.680597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:50.680388    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:50.887866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:50.887829    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:52.108965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:52.108929    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:52.398038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:52.398004    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:52.565896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:52.565861    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:52.884626 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:15:52.884579    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:15:54.105167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:54.105135    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:54.393258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:54.393186    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:54.552707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:54.552662    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:55.886791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:55.886759    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:57.488622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:57.488586    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:59.169619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:59.169581    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:15:59.796466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:15:59.796428    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:00.146290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:00.146238    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:00.633262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:00.633210    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:00.952700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:00.952561    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:02.654937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:02.654897    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:04.358559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:04.358511    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:04.774179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:04.774135    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:04.917374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:04.917332    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:05.340076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.340033    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:05.739324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.739290    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"]
Nov 02 15:16:05.740896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.740874    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:16:05.741031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.741019    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:16:05.741306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.741295    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a2c650-c22a-4b4e-a736-5fc1e23e136f" containerName="cleanup"
Nov 02 15:16:05.741404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.741396    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0569616b-19d3-49b0-ac1d-58fc40dd61d8" containerName="cleanup"
Nov 02 15:16:05.745626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.745607    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.781944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.781916    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c7afd1a6-openshift-default-dockercfg-6t7x9\""
Nov 02 15:16:05.783370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.783345    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"]
Nov 02 15:16:05.825546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.825511    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:05.859155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859118    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istiod-ca-cert\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.859344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859221    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-token\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.859404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859376    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-envoy\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.859465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859433    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-certs\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.859598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859577    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kq9\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-kube-api-access-p2kq9\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.859710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859684    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-data\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.859798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859775    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-credential-socket\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.859844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.859811    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-podinfo\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.860097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.860075    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-socket\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964227    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-socket\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.963774    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-socket\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964436    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istiod-ca-cert\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964489    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-token\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964625    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-envoy\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964723    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-certs\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964826    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kq9\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-kube-api-access-p2kq9\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964886    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-data\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964937    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-credential-socket\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.964960    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-podinfo\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.966853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.966475    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-certs\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.975023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.974992    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-data\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.991075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.975754    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istiod-ca-cert\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.992679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.976008    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-credential-socket\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:05.993884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.993838    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:06.001903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:05.999793    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-envoy\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:06.022551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.022517    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-podinfo\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:06.023663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.023572    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kq9\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-kube-api-access-p2kq9\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:06.026178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.026154    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-token\") pod \"e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") " pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:06.085971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.085930    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:06.453423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.453206    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:06.639474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.639093    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"]
Nov 02 15:16:06.641540 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:16:06.640950    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30f3e5b_5f21_4f8e_ab39_3072f804a77d.slice/crio-bce0a10732ad89ff9c28b521d8b3e342c728ccd2563d7bfb6c2cb782d50f0ceb WatchSource:0}: Error finding container bce0a10732ad89ff9c28b521d8b3e342c728ccd2563d7bfb6c2cb782d50f0ceb: Status 404 returned error can't find the container with id bce0a10732ad89ff9c28b521d8b3e342c728ccd2563d7bfb6c2cb782d50f0ceb
Nov 02 15:16:06.644117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.644078    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:06.644262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.644169    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:06.644262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.644219    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:16:06.720180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.718194    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:06.867616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:06.867572    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:07.333823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:07.332236    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" event={"ID":"d30f3e5b-5f21-4f8e-ab39-3072f804a77d","Type":"ContainerStarted","Data":"864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412"}
Nov 02 15:16:07.333823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:07.332351    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" event={"ID":"d30f3e5b-5f21-4f8e-ab39-3072f804a77d","Type":"ContainerStarted","Data":"bce0a10732ad89ff9c28b521d8b3e342c728ccd2563d7bfb6c2cb782d50f0ceb"}
Nov 02 15:16:07.422673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:07.420340    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podStartSLOduration=2.420319763 podStartE2EDuration="2.420319763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:07.412054517 +0000 UTC m=+1161.666008163" watchObservedRunningTime="2026-11-02 15:16:07.420319763 +0000 UTC m=+1161.674273390"
Nov 02 15:16:08.086415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:08.086362    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:08.092707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:08.092679    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:08.300549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:08.300507    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:08.343765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:08.343642    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:08.344729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:08.344639    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:16:09.955088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:09.955049    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:10.323905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:10.323806    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:10.526077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:10.526042    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:10.730323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:10.730293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:11.628108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:11.628064    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:11.782137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:11.782099    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:12.752877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:12.752840    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:12.904734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:12.904697    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:13.046338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:13.046223    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:14.305252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:14.305213    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:14.450542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:14.450510    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:14.819632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:14.819592    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:16.598343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:16.598305    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:18.374139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:18.374102    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:20.083791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:20.083755    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:21.787524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:21.787481    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:22.706751 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:16:22.706695    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:16:23.378956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:23.378917    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:24.914582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:24.914542    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:26.578291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:26.578253    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:28.240663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:28.240581    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:28.420425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:28.420387    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:30.162003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:30.161957    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:31.889371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:31.889330    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:33.503049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:33.503008    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:35.058190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:35.058132    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:35.201835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:35.201749    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:36.817108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.817073    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"]
Nov 02 15:16:36.821563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.821538    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:36.827368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.827337    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:36.864237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.864191    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:16:36.864435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.864314    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:16:36.875251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.875215    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"]
Nov 02 15:16:36.923019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.922978    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:36.923585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.923554    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:36.924050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:36.924029    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlljv\" (UniqueName: \"kubernetes.io/projected/2add2e6e-c0bf-4799-a999-7f036a67eaac-kube-api-access-wlljv\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.026559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.026520    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.026751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.026618    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlljv\" (UniqueName: \"kubernetes.io/projected/2add2e6e-c0bf-4799-a999-7f036a67eaac-kube-api-access-wlljv\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.026964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.026940    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.027297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.027280    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.029266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.029239    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.052296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.052267    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlljv\" (UniqueName: \"kubernetes.io/projected/2add2e6e-c0bf-4799-a999-7f036a67eaac-kube-api-access-wlljv\") pod \"maas-api-revoke-sub-8c8883b2-q5khd\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.155245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.155208    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:37.478938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.478899    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"]
Nov 02 15:16:37.652789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:37.652756    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd" event={"ID":"2add2e6e-c0bf-4799-a999-7f036a67eaac","Type":"ContainerStarted","Data":"88a84bb20a8c29a7788d246564012cf4d9372b819d1154405b289b74af1130e9"}
Nov 02 15:16:38.426670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:38.424525    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:38.683635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:38.683533    7019 generic.go:356] "Generic (PLEG): container finished" podID="2add2e6e-c0bf-4799-a999-7f036a67eaac" containerID="f2e52fc105712d1b946e81f94ac1906adefd7272c34e8be5737507c6f95092ee" exitCode=0
Nov 02 15:16:38.683635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:38.683621    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd" event={"ID":"2add2e6e-c0bf-4799-a999-7f036a67eaac","Type":"ContainerDied","Data":"f2e52fc105712d1b946e81f94ac1906adefd7272c34e8be5737507c6f95092ee"}
Nov 02 15:16:39.870410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.870384    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:39.985745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.985622    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-xks-ca\") pod \"2add2e6e-c0bf-4799-a999-7f036a67eaac\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") "
Nov 02 15:16:39.985917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.985854    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-openshift-ca\") pod \"2add2e6e-c0bf-4799-a999-7f036a67eaac\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") "
Nov 02 15:16:39.986049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.986031    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2add2e6e-c0bf-4799-a999-7f036a67eaac-kube-api-access-wlljv\" (UniqueName: \"kubernetes.io/projected/2add2e6e-c0bf-4799-a999-7f036a67eaac-kube-api-access-wlljv\") pod \"2add2e6e-c0bf-4799-a999-7f036a67eaac\" (UID: \"2add2e6e-c0bf-4799-a999-7f036a67eaac\") "
Nov 02 15:16:39.986274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.986245    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-openshift-ca" pod "2add2e6e-c0bf-4799-a999-7f036a67eaac" (UID: "2add2e6e-c0bf-4799-a999-7f036a67eaac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:39.987041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.987016    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:16:39.987908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.987878    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-xks-ca" pod "2add2e6e-c0bf-4799-a999-7f036a67eaac" (UID: "2add2e6e-c0bf-4799-a999-7f036a67eaac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:16:39.988641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:39.988599    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2add2e6e-c0bf-4799-a999-7f036a67eaac-kube-api-access-wlljv" pod "2add2e6e-c0bf-4799-a999-7f036a67eaac" (UID: "2add2e6e-c0bf-4799-a999-7f036a67eaac"). InnerVolumeSpecName "kube-api-access-wlljv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:40.088188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:40.088154    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlljv\" (UniqueName: \"kubernetes.io/projected/2add2e6e-c0bf-4799-a999-7f036a67eaac-kube-api-access-wlljv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:16:40.088188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:40.088185    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2add2e6e-c0bf-4799-a999-7f036a67eaac-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:16:40.220707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:40.220622    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:40.702474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:40.702441    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"
Nov 02 15:16:40.702474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:40.702459    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd" event={"ID":"2add2e6e-c0bf-4799-a999-7f036a67eaac","Type":"ContainerDied","Data":"88a84bb20a8c29a7788d246564012cf4d9372b819d1154405b289b74af1130e9"}
Nov 02 15:16:40.702749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:40.702523    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88a84bb20a8c29a7788d246564012cf4d9372b819d1154405b289b74af1130e9"
Nov 02 15:16:41.387314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:41.387249    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:41.912553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:41.912521    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:42.762195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:42.762152    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:43.036248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:43.036157    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"]
Nov 02 15:16:43.049265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:43.049219    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c3b254-zjrgv"]
Nov 02 15:16:43.255445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:43.255400    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:43.417260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:43.417226    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:43.570618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:43.570586    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:43.873718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:43.870742    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6497238-fc86-419e-b767-94bb8618ff96" path="/var/lib/kubelet/pods/a6497238-fc86-419e-b767-94bb8618ff96/volumes"
Nov 02 15:16:45.331067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:45.331007    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:45.660212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:45.660178    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:47.010026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:47.009990    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:47.197891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:47.193735    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:47.374063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:47.374025    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:48.715163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:48.715126    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:48.875102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:48.875067    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:49.049454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:49.049305    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:50.805717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:50.805675    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:50.821633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:50.821595    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:16:52.710815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:52.710776    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:52.731354 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:16:52.731319    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:16:54.433672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:54.425277    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:54.584676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:54.584457    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:54.873745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:54.873130    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:55.037480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:55.033680    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:56.061196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:56.059712    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"]
Nov 02 15:16:56.073958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:56.073887    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2d95328-q8qbr"]
Nov 02 15:16:56.514601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:56.514569    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:57.863043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:57.863012    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="652c69ed-2903-4c5e-a33b-71458c773222" path="/var/lib/kubelet/pods/652c69ed-2903-4c5e-a33b-71458c773222/volumes"
Nov 02 15:16:58.305207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:58.305117    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:16:59.927552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:16:59.927511    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:00.099016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:00.098981    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:00.805849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:00.805808    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:01.282883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.282787    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:01.689749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.689716    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:01.705845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.705796    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"]
Nov 02 15:17:01.707495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.707461    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2add2e6e-c0bf-4799-a999-7f036a67eaac" containerName="revoke-keys"
Nov 02 15:17:01.707686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.707670    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2add2e6e-c0bf-4799-a999-7f036a67eaac" containerName="revoke-keys"
Nov 02 15:17:01.707958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.707943    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2add2e6e-c0bf-4799-a999-7f036a67eaac" containerName="revoke-keys"
Nov 02 15:17:01.733024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.732972    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"]
Nov 02 15:17:01.733217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.733099    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.790053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.778113    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0a7163-openshift-default-dockercfg-m9vkd\""
Nov 02 15:17:01.860595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.860555    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-podinfo\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.860932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.860907    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istiod-ca-cert\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.861190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.861169    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-data\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.861381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.861364    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-token\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.861494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.861477    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrckf\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-kube-api-access-rrckf\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.861693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.861636    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-socket\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.861887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.861869    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-certs\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.862089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.862058    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-credential-socket\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.862390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.862191    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-envoy\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.875566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.875393    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:01.969324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.969226    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-certs\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.969324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.964430    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-certs\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.970015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.969986    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-credential-socket\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.975108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.975071    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-credential-socket\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.975340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.975321    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-envoy\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.975558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.975543    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-podinfo\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.984900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.975748    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istiod-ca-cert\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.985223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.978112    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istiod-ca-cert\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.987278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.985612    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-data\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.987278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.985761    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-token\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.987278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.985789    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrckf\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-kube-api-access-rrckf\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.987278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.985864    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-socket\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.987278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.986376    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-socket\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:01.987278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:01.986613    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-data\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:02.008326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.008237    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-envoy\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:02.013087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.013025    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-podinfo\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:02.016558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.016492    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrckf\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-kube-api-access-rrckf\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:02.055585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.055480    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-token\") pod \"e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") " pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:02.140685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.140326    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:02.750014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.749980    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"]
Nov 02 15:17:02.751710 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:17:02.751630    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e68d4b4_3616_40af_86bc_d419e0d0fc0e.slice/crio-a4edfd3db32a3a871cb0a98caad77cefc2743efcf9324b73106bd5620185484b WatchSource:0}: Error finding container a4edfd3db32a3a871cb0a98caad77cefc2743efcf9324b73106bd5620185484b: Status 404 returned error can't find the container with id a4edfd3db32a3a871cb0a98caad77cefc2743efcf9324b73106bd5620185484b
Nov 02 15:17:02.755306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.755063    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:02.755306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.755162    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:02.755306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:02.755199    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:03.011088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.009149    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" event={"ID":"9e68d4b4-3616-40af-86bc-d419e0d0fc0e","Type":"ContainerStarted","Data":"90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243"}
Nov 02 15:17:03.011088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.009219    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" event={"ID":"9e68d4b4-3616-40af-86bc-d419e0d0fc0e","Type":"ContainerStarted","Data":"a4edfd3db32a3a871cb0a98caad77cefc2743efcf9324b73106bd5620185484b"}
Nov 02 15:17:03.063706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.063266    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podStartSLOduration=2.063246319 podStartE2EDuration="2.063246319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:03.060491059 +0000 UTC m=+1217.314444707" watchObservedRunningTime="2026-11-02 15:17:03.063246319 +0000 UTC m=+1217.317199943"
Nov 02 15:17:03.141967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.141925    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:03.143132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.143103    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:17:03.143278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.143152    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 15:17:03.485722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.485063    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:03.748796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.748085    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:03.964531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.964491    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:17:03.964997 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:03.964559    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 02 15:17:04.022159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:04.022047    7019 generic.go:356] "Generic (PLEG): container finished" podID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerID="1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda" exitCode=0
Nov 02 15:17:04.022159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:04.022116    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" event={"ID":"93149e3b-33b5-4cce-bbec-972ffd4b1c61","Type":"ContainerDied","Data":"1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda"}
Nov 02 15:17:04.023074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:04.023051    7019 scope.go:122] "RemoveContainer" containerID="1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda"
Nov 02 15:17:04.025311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:04.025271    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:04.025430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:04.025364    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:04.025430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:04.025408    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:04.149291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:04.149259    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:05.032716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.032679    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" event={"ID":"93149e3b-33b5-4cce-bbec-972ffd4b1c61","Type":"ContainerStarted","Data":"781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe"}
Nov 02 15:17:05.034370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.034349    7019 generic.go:356] "Generic (PLEG): container finished" podID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerID="864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412" exitCode=0
Nov 02 15:17:05.034493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.034425    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" event={"ID":"d30f3e5b-5f21-4f8e-ab39-3072f804a77d","Type":"ContainerDied","Data":"864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412"}
Nov 02 15:17:05.034860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.034837    7019 scope.go:122] "RemoveContainer" containerID="864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412"
Nov 02 15:17:05.034996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.034861    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:05.036100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.036082    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:17:05.036621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.036594    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:05.036747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.036721    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:05.036805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.036783    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:05.048699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.048674    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:05.557891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.557851    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:17:05.558080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.557896    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:17:05.558080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.557992    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 02 15:17:05.711312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.711226    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:05.944160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:05.938336    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:06.092043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:06.091999    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" event={"ID":"d30f3e5b-5f21-4f8e-ab39-3072f804a77d","Type":"ContainerStarted","Data":"7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607"}
Nov 02 15:17:06.144687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:06.142747    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:06.400415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:06.400377    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:06.559111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:06.559073    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:17:06.559394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:06.559365    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 02 15:17:06.602782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:06.602748    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:07.028204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.028164    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:07.086864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.086809    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:17:07.088565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.088536    7019 patch_prober.go:29] interesting pod/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:17:07.088885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.088846    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 15:17:07.252911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.252873    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:07.557958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.557920    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:17:07.558121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.557987    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 02 15:17:07.718674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:07.718114    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:08.097175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.097122    7019 patch_prober.go:29] interesting pod/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:17:08.097843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.097461    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 15:17:08.129676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.119610    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"]
Nov 02 15:17:08.136106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.136078    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.140102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.138888    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"]
Nov 02 15:17:08.153100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.153054    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:17:08.153248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.153236    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:17:08.251894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.251857    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.252069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.251985    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhz9\" (UniqueName: \"kubernetes.io/projected/e63ef6d7-4a05-41a8-b19f-13be95f965dd-kube-api-access-5bhz9\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.252271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.252249    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.352607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.349251    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:08.352607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.349614    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:17:08.356373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.356134    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.359760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.358127    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.359760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.358262    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhz9\" (UniqueName: \"kubernetes.io/projected/e63ef6d7-4a05-41a8-b19f-13be95f965dd-kube-api-access-5bhz9\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.359760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.359134    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.364160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.364124    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.392673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.391131    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhz9\" (UniqueName: \"kubernetes.io/projected/e63ef6d7-4a05-41a8-b19f-13be95f965dd-kube-api-access-5bhz9\") pod \"maas-api-revoke-sub-b253fd2f-xppqp\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.466687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.458394    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:08.558488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.558459    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:17:08.558625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.558511    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 02 15:17:08.947795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:08.947735    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"]
Nov 02 15:17:08.952373 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:17:08.952337    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63ef6d7_4a05_41a8_b19f_13be95f965dd.slice/crio-8636740a0ba4612f1c8af396ef735b6d216405d46b4683958a85520fcf9bc5e4 WatchSource:0}: Error finding container 8636740a0ba4612f1c8af396ef735b6d216405d46b4683958a85520fcf9bc5e4: Status 404 returned error can't find the container with id 8636740a0ba4612f1c8af396ef735b6d216405d46b4683958a85520fcf9bc5e4
Nov 02 15:17:09.114673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:09.109788    7019 patch_prober.go:29] interesting pod/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:17:09.114673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:09.109854    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 15:17:09.151404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:09.151371    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp" event={"ID":"e63ef6d7-4a05-41a8-b19f-13be95f965dd","Type":"ContainerStarted","Data":"8636740a0ba4612f1c8af396ef735b6d216405d46b4683958a85520fcf9bc5e4"}
Nov 02 15:17:09.559093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:09.559060    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:17:09.559455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:09.559118    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 02 15:17:10.090672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.090170    7019 patch_prober.go:29] interesting pod/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:17:10.090672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.090238    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 15:17:10.315196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.315156    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:10.442795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.441426    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp" event={"ID":"e63ef6d7-4a05-41a8-b19f-13be95f965dd","Type":"ContainerStarted","Data":"b38886e083995b4c8dfddc2f031edb8521fc454b182cd7b80e6b451d82741d4e"}
Nov 02 15:17:10.572764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.569055    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 02 15:17:10.572764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.569118    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 02 15:17:10.628679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.627574    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:10.700837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:10.700296    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp" podStartSLOduration=2.7002732480000002 podStartE2EDuration="2.700273248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:10.540961315 +0000 UTC m=+1224.794914940" watchObservedRunningTime="2026-11-02 15:17:10.700273248 +0000 UTC m=+1224.954226872"
Nov 02 15:17:11.089731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.089010    7019 patch_prober.go:29] interesting pod/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:17:11.089731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.089090    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 15:17:11.374039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.374007    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:11.453292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.453261    7019 generic.go:356] "Generic (PLEG): container finished" podID="e63ef6d7-4a05-41a8-b19f-13be95f965dd" containerID="b38886e083995b4c8dfddc2f031edb8521fc454b182cd7b80e6b451d82741d4e" exitCode=0
Nov 02 15:17:11.453492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.453297    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp" event={"ID":"e63ef6d7-4a05-41a8-b19f-13be95f965dd","Type":"ContainerDied","Data":"b38886e083995b4c8dfddc2f031edb8521fc454b182cd7b80e6b451d82741d4e"}
Nov 02 15:17:11.563204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.563175    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:17:11.563726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.563703    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:17:11.564331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:11.564313    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:17:12.089084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.089044    7019 patch_prober.go:29] interesting pod/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 15:17:12.089784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.089745    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 15:17:12.261919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.261881    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:12.663697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.663638    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:12.746886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.746786    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-xks-ca\") pod \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") "
Nov 02 15:17:12.746886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.746890    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63ef6d7-4a05-41a8-b19f-13be95f965dd-kube-api-access-5bhz9\" (UniqueName: \"kubernetes.io/projected/e63ef6d7-4a05-41a8-b19f-13be95f965dd-kube-api-access-5bhz9\") pod \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") "
Nov 02 15:17:12.747130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.746929    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-openshift-ca\") pod \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\" (UID: \"e63ef6d7-4a05-41a8-b19f-13be95f965dd\") "
Nov 02 15:17:12.747353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.747324    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-openshift-ca" pod "e63ef6d7-4a05-41a8-b19f-13be95f965dd" (UID: "e63ef6d7-4a05-41a8-b19f-13be95f965dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:12.747800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.747782    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:12.749400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.749372    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-xks-ca" pod "e63ef6d7-4a05-41a8-b19f-13be95f965dd" (UID: "e63ef6d7-4a05-41a8-b19f-13be95f965dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:17:12.749590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.749574    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63ef6d7-4a05-41a8-b19f-13be95f965dd-kube-api-access-5bhz9" pod "e63ef6d7-4a05-41a8-b19f-13be95f965dd" (UID: "e63ef6d7-4a05-41a8-b19f-13be95f965dd"). InnerVolumeSpecName "kube-api-access-5bhz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:12.849224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.849182    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhz9\" (UniqueName: \"kubernetes.io/projected/e63ef6d7-4a05-41a8-b19f-13be95f965dd-kube-api-access-5bhz9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:12.849224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:12.849217    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e63ef6d7-4a05-41a8-b19f-13be95f965dd-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:13.091744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:13.091670    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:17:13.092982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:13.092962    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:17:13.531694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:13.521769    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"
Nov 02 15:17:13.531694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:13.522142    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp" event={"ID":"e63ef6d7-4a05-41a8-b19f-13be95f965dd","Type":"ContainerDied","Data":"8636740a0ba4612f1c8af396ef735b6d216405d46b4683958a85520fcf9bc5e4"}
Nov 02 15:17:13.531694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:13.522321    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8636740a0ba4612f1c8af396ef735b6d216405d46b4683958a85520fcf9bc5e4"
Nov 02 15:17:13.962920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:13.962882    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:14.225221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:14.225130    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:15.693787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.693745    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:15.795330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.795294    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"]
Nov 02 15:17:15.796086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.796067    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e63ef6d7-4a05-41a8-b19f-13be95f965dd" containerName="revoke-keys"
Nov 02 15:17:15.796136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.796087    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63ef6d7-4a05-41a8-b19f-13be95f965dd" containerName="revoke-keys"
Nov 02 15:17:15.796246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.796237    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63ef6d7-4a05-41a8-b19f-13be95f965dd" containerName="revoke-keys"
Nov 02 15:17:15.799418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.799398    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.821478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.821440    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl2zj\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-kube-api-access-hl2zj\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.821718    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55f9fad-b571-44b9-97ea-145f877a4905-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.821782    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.821818    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.821865    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.821915    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.821962    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55f9fad-b571-44b9-97ea-145f877a4905-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.822035    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.822909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.822196    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.835067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.835016    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-246137-openshift-default-dockercfg-hmh98\""
Nov 02 15:17:15.837428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.837381    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"]
Nov 02 15:17:15.867230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.866979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925360    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925588    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl2zj\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-kube-api-access-hl2zj\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925641    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55f9fad-b571-44b9-97ea-145f877a4905-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925702    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925740    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925783    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925834    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925900    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55f9fad-b571-44b9-97ea-145f877a4905-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.926335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.925972    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.927414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.927389    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.927564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.927531    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.927736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.927428    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.927880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.927751    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.939732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.937681    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55f9fad-b571-44b9-97ea-145f877a4905-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.939732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.937814    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.940680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.940629    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55f9fad-b571-44b9-97ea-145f877a4905-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.953909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.953839    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:15.954714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:15.954663    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl2zj\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-kube-api-access-hl2zj\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:16.113173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.113136    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:16.468046 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:17:16.468014    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55f9fad_b571_44b9_97ea_145f877a4905.slice/crio-f4667c599aae77ffcff2aef9fa399faf30fd4694dcc32d6a5a670232dc8acf55 WatchSource:0}: Error finding container f4667c599aae77ffcff2aef9fa399faf30fd4694dcc32d6a5a670232dc8acf55: Status 404 returned error can't find the container with id f4667c599aae77ffcff2aef9fa399faf30fd4694dcc32d6a5a670232dc8acf55
Nov 02 15:17:16.468171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.468133    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"]
Nov 02 15:17:16.471254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.471204    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:16.471503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.471457    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:16.471676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.471637    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:16.561464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.561429    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" event={"ID":"e55f9fad-b571-44b9-97ea-145f877a4905","Type":"ContainerStarted","Data":"f4667c599aae77ffcff2aef9fa399faf30fd4694dcc32d6a5a670232dc8acf55"}
Nov 02 15:17:16.885681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.885112    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"]
Nov 02 15:17:16.889568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.889528    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:16.905278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.900106    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"]
Nov 02 15:17:16.914479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.914442    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3d7df16e-openshift-default-dockercfg-sn9kd\""
Nov 02 15:17:16.968972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.968940    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:16.969341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:16.969300    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.015350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.015303    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896jc\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-kube-api-access-896jc\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.030722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.030685    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.031030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.031011    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.031203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.031153    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.031370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.031358    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.031556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.031533    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.031635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.031607    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.136700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.136573    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.136700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.136692    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.136937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.136810    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.136937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.136871    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.136981    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896jc\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-kube-api-access-896jc\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.137013    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.137052    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.137074    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.137151    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.137419    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.137641    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.137788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.137729    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.138547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.138516    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.138658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.138626    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.141007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.140966    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.141298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.141277    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.166238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.166203    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.183141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.183110    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-896jc\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-kube-api-access-896jc\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.314106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.314060    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:17.654259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.654210    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" event={"ID":"e55f9fad-b571-44b9-97ea-145f877a4905","Type":"ContainerStarted","Data":"d6cd9fb72fb3721c9313f4ba52856c897e44f2b36a0b21dcbabd0c9139979613"}
Nov 02 15:17:17.727265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.727191    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" podStartSLOduration=2.727168133 podStartE2EDuration="2.727168133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:17.708174753 +0000 UTC m=+1231.962128378" watchObservedRunningTime="2026-11-02 15:17:17.727168133 +0000 UTC m=+1231.981121760"
Nov 02 15:17:17.927386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:17.927347    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:18.016672 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:17:18.013101    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eaf17e3_6c7b_4dc7_a76d_ba88934916e3.slice/crio-eadebf56c6f894100e03ca455fe9ba460671caff206d7d9267250987564e3e31 WatchSource:0}: Error finding container eadebf56c6f894100e03ca455fe9ba460671caff206d7d9267250987564e3e31: Status 404 returned error can't find the container with id eadebf56c6f894100e03ca455fe9ba460671caff206d7d9267250987564e3e31
Nov 02 15:17:18.026781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.026673    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"]
Nov 02 15:17:18.031549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.030365    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:18.031549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.030485    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:18.031549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.030589    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:18.113834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.113795    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:18.115219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.115188    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 15:17:18.115475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.115446    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 15:17:18.583726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.577694    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:18.670681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.668722    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" event={"ID":"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3","Type":"ContainerStarted","Data":"5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22"}
Nov 02 15:17:18.670681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.668791    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" event={"ID":"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3","Type":"ContainerStarted","Data":"eadebf56c6f894100e03ca455fe9ba460671caff206d7d9267250987564e3e31"}
Nov 02 15:17:18.723266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:18.723182    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podStartSLOduration=2.723162295 podStartE2EDuration="2.723162295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:18.720426112 +0000 UTC m=+1232.974379741" watchObservedRunningTime="2026-11-02 15:17:18.723162295 +0000 UTC m=+1232.977115920"
Nov 02 15:17:19.122011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:19.114079    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 15:17:19.122011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:19.114144    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 15:17:19.319710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:19.319214    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:19.321478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:19.321444    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 15:17:19.321733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:19.321703    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 15:17:19.845275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:19.842358    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:20.145680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:20.139865    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.145680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:20.139930    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 15:17:20.320064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:20.320015    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 15:17:20.320274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:20.320094    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 15:17:21.117673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:21.117140    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.117673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:21.117212    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 15:17:21.315670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:21.315098    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 15:17:21.315670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:21.315165    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 15:17:21.704675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:21.702359    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:22.136716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:22.114150    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.137025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:22.136981    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Nov 02 15:17:22.315148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:22.315107    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 15:17:22.317345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:22.317309    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 15:17:22.846231 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:17:22.840535    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:17:23.197676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:23.182986    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:23.197676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:23.186978    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:23.200711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:23.200616    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:17:23.315766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:23.315730    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 15:17:23.315936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:23.315793    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 15:17:23.978512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:23.978475    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:24.316139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:24.316045    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 15:17:24.316391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:24.316362    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 15:17:25.321032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:25.320996    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:25.321473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:25.321414    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:25.322421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:25.322398    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:17:25.907182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:25.907137    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:26.113706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:26.113675    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:27.602947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:27.602910    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:29.154940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:29.154888    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:30.758048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:30.758012    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:32.414749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:32.414709    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:34.085669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:34.083793    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:17:34.113669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:34.110299    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:36.370944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:36.370902    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:36.521441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:36.521288    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:36.952669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:36.952309    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:37.795805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:37.795756    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:37.991672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:37.987545    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:17:38.017832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:38.017776    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:17:38.778937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:38.778904    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:40.554348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:40.554309    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:40.918762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:40.918724    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"]
Nov 02 15:17:40.918980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:40.918945    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy" containerID="cri-o://5226135c192c91ac56cffdbab53b0f1c7adf80623e4d3d5bd540530ad0044a2c" gracePeriod=30
Nov 02 15:17:40.936113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:40.936085    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"]
Nov 02 15:17:40.936314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:40.936274    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy" containerID="cri-o://9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da" gracePeriod=30
Nov 02 15:17:40.942496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:40.942468    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"]
Nov 02 15:17:40.942744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:40.942716    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy" containerID="cri-o://3d32dbd7d6d6df1cbeb2029592e7614bc528c49487b0e3cdab58ef9d69564322" gracePeriod=30
Nov 02 15:17:42.480427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:42.480385    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:43.403798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.403759    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"]
Nov 02 15:17:43.409067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.409041    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.430616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.430582    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"]
Nov 02 15:17:43.430842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.430818    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ad276f-openshift-default-dockercfg-s8d7n\""
Nov 02 15:17:43.464978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.464941    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-podinfo\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465058    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465124    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istiod-ca-cert\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465145    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-envoy\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465298    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-credential-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465353    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-certs\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465475    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-token\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465579    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-data\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.465699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.465680    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5zn\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-kube-api-access-7w5zn\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.567889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.567849    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-data\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.567926    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5zn\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-kube-api-access-7w5zn\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568132    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-podinfo\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568184    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568229    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istiod-ca-cert\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568255    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-envoy\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568357    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-credential-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568417    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-certs\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568473    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-token\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568567    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.568883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568348    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-data\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.569572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.568939    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-certs\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.569572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.569041    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istiod-ca-cert\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.569572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.569168    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-credential-socket\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.570870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.570829    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-envoy\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.571162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.571140    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-podinfo\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.596216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.596182    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-token\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.596419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.596396    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5zn\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-kube-api-access-7w5zn\") pod \"e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") " pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:43.628082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.628027    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:43.725138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:43.725047    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:44.059419 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:17:44.059386    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edb6e4b_d183_4ae9_9f47_7c68c8dce8c0.slice/crio-9e8cf5aff3560efd87f0b540d412280956c44447cdb34f54181aaaade5362993 WatchSource:0}: Error finding container 9e8cf5aff3560efd87f0b540d412280956c44447cdb34f54181aaaade5362993: Status 404 returned error can't find the container with id 9e8cf5aff3560efd87f0b540d412280956c44447cdb34f54181aaaade5362993
Nov 02 15:17:44.059946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.059924    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"]
Nov 02 15:17:44.062308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.062274    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:44.062392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.062340    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:44.062392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.062373    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:17:44.086854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.086822    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:44.243498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.243441    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" event={"ID":"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0","Type":"ContainerStarted","Data":"72ef6e99d8b71199f37a1d5ea684667214857f35ead9d0543d281ce54cd3d393"}
Nov 02 15:17:44.243794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.243528    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" event={"ID":"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0","Type":"ContainerStarted","Data":"9e8cf5aff3560efd87f0b540d412280956c44447cdb34f54181aaaade5362993"}
Nov 02 15:17:44.310566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.310453    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podStartSLOduration=1.310430928 podStartE2EDuration="1.310430928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:44.306296682 +0000 UTC m=+1258.560250306" watchObservedRunningTime="2026-11-02 15:17:44.310430928 +0000 UTC m=+1258.564384554"
Nov 02 15:17:44.725727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:44.725695    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:45.727296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:45.727259    7019 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:17:45.727722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:45.727313    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 15:17:45.781675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:45.779936    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:45.928182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:45.928135    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:46.429047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.429008    7019 generic.go:356] "Generic (PLEG): container finished" podID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerID="9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da" exitCode=0
Nov 02 15:17:46.429488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.429457    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" event={"ID":"e94d8223-7b91-4d72-9073-bfc6afa6888c","Type":"ContainerDied","Data":"9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da"}
Nov 02 15:17:46.432142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.432114    7019 generic.go:356] "Generic (PLEG): container finished" podID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerID="3d32dbd7d6d6df1cbeb2029592e7614bc528c49487b0e3cdab58ef9d69564322" exitCode=0
Nov 02 15:17:46.432555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.432528    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" event={"ID":"94546aca-dcfe-441c-a14c-16735eb4bc00","Type":"ContainerDied","Data":"3d32dbd7d6d6df1cbeb2029592e7614bc528c49487b0e3cdab58ef9d69564322"}
Nov 02 15:17:46.435580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.435540    7019 generic.go:356] "Generic (PLEG): container finished" podID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerID="5226135c192c91ac56cffdbab53b0f1c7adf80623e4d3d5bd540530ad0044a2c" exitCode=0
Nov 02 15:17:46.435712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.435613    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" event={"ID":"398f9eb8-3c21-4f44-8172-bd99b4618c40","Type":"ContainerDied","Data":"5226135c192c91ac56cffdbab53b0f1c7adf80623e4d3d5bd540530ad0044a2c"}
Nov 02 15:17:46.559120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.559084    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:17:46.648409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.648382    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:17:46.653377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.653357    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:17:46.737967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.737863    7019 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:17:46.738525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.738481    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 15:17:46.745931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.745898    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-data\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.745959    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-token\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-token\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.746085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.745991    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94546aca-dcfe-441c-a14c-16735eb4bc00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94546aca-dcfe-441c-a14c-16735eb4bc00-istiod-ca-cert\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746034    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398f9eb8-3c21-4f44-8172-bd99b4618c40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398f9eb8-3c21-4f44-8172-bd99b4618c40-istiod-ca-cert\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.746085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746068    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-credential-socket\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746126    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-socket\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746164    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-podinfo\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.746300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746237    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-certs\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746303    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-envoy\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.746455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746332    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-podinfo\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746367    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-token\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-token\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746399    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-credential-socket\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.746455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746434    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-socket\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.746721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746471    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94d8223-7b91-4d72-9073-bfc6afa6888c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d8223-7b91-4d72-9073-bfc6afa6888c-istiod-ca-cert\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.746721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746514    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-envoy\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.746721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746570    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-kube-api-access-bx7qz\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-kube-api-access-bx7qz\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.746721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746609    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-data\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.746721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746643    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-certs\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.746721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746714    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-socket\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.747025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746773    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-podinfo\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.747025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746863    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-token\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-token\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.747025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746925    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-certs\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.747025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.746959    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-kube-api-access-zhhbf\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-kube-api-access-zhhbf\") pod \"94546aca-dcfe-441c-a14c-16735eb4bc00\" (UID: \"94546aca-dcfe-441c-a14c-16735eb4bc00\") "
Nov 02 15:17:46.747025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.747012    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-kube-api-access-h59lt\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-kube-api-access-h59lt\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.747268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.747085    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-data\") pod \"e94d8223-7b91-4d72-9073-bfc6afa6888c\" (UID: \"e94d8223-7b91-4d72-9073-bfc6afa6888c\") "
Nov 02 15:17:46.747268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.747125    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-credential-socket\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.747268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.747262    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-envoy\") pod \"398f9eb8-3c21-4f44-8172-bd99b4618c40\" (UID: \"398f9eb8-3c21-4f44-8172-bd99b4618c40\") "
Nov 02 15:17:46.748787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.748739    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-socket" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.749125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.748826    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-socket" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.749246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.749010    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-data" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.749342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.749316    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398f9eb8-3c21-4f44-8172-bd99b4618c40-istiod-ca-cert" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:46.749603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.749579    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-credential-socket" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.749847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.749825    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94546aca-dcfe-441c-a14c-16735eb4bc00-istiod-ca-cert" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:46.750770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.750739    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-credential-socket" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.751059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.751034    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-certs" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.753466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.753441    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94d8223-7b91-4d72-9073-bfc6afa6888c-istiod-ca-cert" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:17:46.758899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.757984    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-token" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.758899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.758035    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-podinfo" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:46.758899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.758078    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-token" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.758899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.758110    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-envoy" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.759672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.759485    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-podinfo" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:46.759922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.759805    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-envoy" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.760159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.760042    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-certs" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.760374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.760254    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-certs" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.760549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.760445    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-socket" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.760893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.760627    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-data" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.760893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.760790    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-envoy" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.761065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.760975    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-credential-socket" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.761206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.761165    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-data" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:46.761411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.761345    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-podinfo" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:17:46.761634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.761607    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-kube-api-access-h59lt" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "kube-api-access-h59lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.761783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.761761    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-kube-api-access-zhhbf" pod "94546aca-dcfe-441c-a14c-16735eb4bc00" (UID: "94546aca-dcfe-441c-a14c-16735eb4bc00"). InnerVolumeSpecName "kube-api-access-zhhbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.761947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.761917    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-token" pod "e94d8223-7b91-4d72-9073-bfc6afa6888c" (UID: "e94d8223-7b91-4d72-9073-bfc6afa6888c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.762021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.762001    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-kube-api-access-bx7qz" pod "398f9eb8-3c21-4f44-8172-bd99b4618c40" (UID: "398f9eb8-3c21-4f44-8172-bd99b4618c40"). InnerVolumeSpecName "kube-api-access-bx7qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:46.850116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850088    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850450    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850557    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850633    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94546aca-dcfe-441c-a14c-16735eb4bc00-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850696    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/398f9eb8-3c21-4f44-8172-bd99b4618c40-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850714    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850728    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850745    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850759    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850773    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850786    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850799    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850813    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.850833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850826    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850841    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e94d8223-7b91-4d72-9073-bfc6afa6888c-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850856    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94546aca-dcfe-441c-a14c-16735eb4bc00-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850870    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx7qz\" (UniqueName: \"kubernetes.io/projected/398f9eb8-3c21-4f44-8172-bd99b4618c40-kube-api-access-bx7qz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850884    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850897    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850911    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850925    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/398f9eb8-3c21-4f44-8172-bd99b4618c40-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850939    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850952    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850965    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhhbf\" (UniqueName: \"kubernetes.io/projected/94546aca-dcfe-441c-a14c-16735eb4bc00-kube-api-access-zhhbf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850979    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h59lt\" (UniqueName: \"kubernetes.io/projected/e94d8223-7b91-4d72-9073-bfc6afa6888c-kube-api-access-h59lt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.850992    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e94d8223-7b91-4d72-9073-bfc6afa6888c-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:46.851150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:46.851006    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/398f9eb8-3c21-4f44-8172-bd99b4618c40-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:17:47.111689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.111659    7019 scope.go:122] "RemoveContainer" containerID="5226135c192c91ac56cffdbab53b0f1c7adf80623e4d3d5bd540530ad0044a2c"
Nov 02 15:17:47.129331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.129306    7019 scope.go:122] "RemoveContainer" containerID="b148a5bfa87dcb1fef9b18246eba96a8b891fd11c8c1c11e6820e282a191bc53"
Nov 02 15:17:47.141761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.141735    7019 scope.go:122] "RemoveContainer" containerID="9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da"
Nov 02 15:17:47.229390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.229248    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:47.488733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.483414    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq" event={"ID":"e94d8223-7b91-4d72-9073-bfc6afa6888c","Type":"ContainerDied","Data":"489e3234b928ffab662053d4b227c90b5da13f1d473c7661e038c555fca1d58d"}
Nov 02 15:17:47.531172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.531120    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:47.533188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.533166    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"
Nov 02 15:17:47.533785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.533755    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"
Nov 02 15:17:47.559687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.555498    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f" event={"ID":"94546aca-dcfe-441c-a14c-16735eb4bc00","Type":"ContainerDied","Data":"d18aeafeaf044e86c4d2950585f66fea46cb329f77b1238c8d14ae8aae9b3e3f"}
Nov 02 15:17:47.559687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.555582    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx" event={"ID":"398f9eb8-3c21-4f44-8172-bd99b4618c40","Type":"ContainerDied","Data":"8d498764486bdfa592718beae20ef70fb4b7941f3dcf1a414e1d75ade3702656"}
Nov 02 15:17:47.559687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.555619    7019 scope.go:122] "RemoveContainer" containerID="3d32dbd7d6d6df1cbeb2029592e7614bc528c49487b0e3cdab58ef9d69564322"
Nov 02 15:17:47.559687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.555766    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"
Nov 02 15:17:47.767456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.762539    7019 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:17:47.767456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.762597    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 15:17:47.971680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.970083    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"]
Nov 02 15:17:47.971680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.970132    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-7cgrq"]
Nov 02 15:17:47.984819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:47.984786    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"]
Nov 02 15:17:48.031743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:48.031503    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-fjpbx"]
Nov 02 15:17:48.048893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:48.048865    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"]
Nov 02 15:17:48.055567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:48.055527    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-n754f"]
Nov 02 15:17:48.130607 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:17:48.130571    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:17:48.726469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:48.726425    7019 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:17:48.726738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:48.726490    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 15:17:49.448966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:49.448928    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:49.725485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:49.725403    7019 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:17:49.725485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:49.725463    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 15:17:49.904641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:49.904605    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" path="/var/lib/kubelet/pods/398f9eb8-3c21-4f44-8172-bd99b4618c40/volumes"
Nov 02 15:17:49.905233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:49.905218    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" path="/var/lib/kubelet/pods/94546aca-dcfe-441c-a14c-16735eb4bc00/volumes"
Nov 02 15:17:49.905751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:49.905736    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" path="/var/lib/kubelet/pods/e94d8223-7b91-4d72-9073-bfc6afa6888c/volumes"
Nov 02 15:17:50.726011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:50.725967    7019 patch_prober.go:29] interesting pod/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 15:17:50.726416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:50.726030    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 15:17:51.176723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:51.176683    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:51.730333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:51.730301    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:51.730870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:51.730676    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:51.731540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:51.731519    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:17:52.707315 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:17:52.707271    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:17:52.870576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:52.870541    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:53.036702 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:17:53.036572    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:17:53.075140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:53.075093    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:53.723583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:53.723549    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:54.190184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:54.190109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:54.352768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:54.352731    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:54.566731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:54.563751    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:55.191910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:55.191875    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"]
Nov 02 15:17:55.192357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:55.192088    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy" containerID="cri-o://f0ad15fd25efb1c7e79b61d609e09c2923e105af4c0a0a8a5fd55a8a00774df1" gracePeriod=30
Nov 02 15:17:55.209394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:55.209362    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"]
Nov 02 15:17:55.209640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:55.209612    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy" containerID="cri-o://6ebdc2e00691930b42a59a9649ae2fb246d661804cd933f0ba0e89139572d5f1" gracePeriod=30
Nov 02 15:17:55.213846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:55.213808    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"]
Nov 02 15:17:55.214059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:55.214034    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy" containerID="cri-o://5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4" gracePeriod=30
Nov 02 15:17:56.518302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:56.518260    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:58.222798 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:17:58.222755    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:17:58.295360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:58.295324    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:58.453001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:58.452969    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:17:58.607573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:17:58.607488    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:00.093015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.092970    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:00.266254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.266213    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:00.622743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.620613    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:00.656362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.655381    7019 generic.go:356] "Generic (PLEG): container finished" podID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerID="f0ad15fd25efb1c7e79b61d609e09c2923e105af4c0a0a8a5fd55a8a00774df1" exitCode=0
Nov 02 15:18:00.656362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.655446    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" event={"ID":"bbeed7de-92f1-4222-b9cd-e7edff50e02c","Type":"ContainerDied","Data":"f0ad15fd25efb1c7e79b61d609e09c2923e105af4c0a0a8a5fd55a8a00774df1"}
Nov 02 15:18:00.657800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.657669    7019 generic.go:356] "Generic (PLEG): container finished" podID="45d61337-0a44-4b54-8259-45b7c4467893" containerID="6ebdc2e00691930b42a59a9649ae2fb246d661804cd933f0ba0e89139572d5f1" exitCode=0
Nov 02 15:18:00.658161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.658116    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" event={"ID":"45d61337-0a44-4b54-8259-45b7c4467893","Type":"ContainerDied","Data":"6ebdc2e00691930b42a59a9649ae2fb246d661804cd933f0ba0e89139572d5f1"}
Nov 02 15:18:00.928086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.928055    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:18:00.955985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.955895    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"]
Nov 02 15:18:00.961581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.961551    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy"
Nov 02 15:18:00.961581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.961580    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy"
Nov 02 15:18:00.961782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.961605    7019 cpu_manager.go:414] "containerMap: removing container" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy"
Nov 02 15:18:00.961782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.961613    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy"
Nov 02 15:18:00.962246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962225    7019 cpu_manager.go:414] "containerMap: removing container" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy"
Nov 02 15:18:00.962353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962272    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy"
Nov 02 15:18:00.962353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962293    7019 cpu_manager.go:414] "containerMap: removing container" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy"
Nov 02 15:18:00.962353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962301    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy"
Nov 02 15:18:00.962522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962488    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="398f9eb8-3c21-4f44-8172-bd99b4618c40" containerName="istio-proxy"
Nov 02 15:18:00.962522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962504    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="94546aca-dcfe-441c-a14c-16735eb4bc00" containerName="istio-proxy"
Nov 02 15:18:00.962621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962529    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94d8223-7b91-4d72-9073-bfc6afa6888c" containerName="istio-proxy"
Nov 02 15:18:00.962621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.962540    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" containerName="istio-proxy"
Nov 02 15:18:00.965339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.965311    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:18:00.969876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.969853    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:00.984166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.984136    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"]
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998152    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-certs\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998204    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-socket\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998248    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-credential-socket\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998276    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-envoy\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998340    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-certs\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998370    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-data\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998501    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-istio-token\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-istio-token\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998528    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istiod-ca-cert\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998562    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-kube-api-access-mvtpn\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-kube-api-access-mvtpn\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998611    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-podinfo\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998668    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d61337-0a44-4b54-8259-45b7c4467893-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d61337-0a44-4b54-8259-45b7c4467893-istiod-ca-cert\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998705    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-credential-socket\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998771    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-data\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998855    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-token\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-token\") pod \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\" (UID: \"bbeed7de-92f1-4222-b9cd-e7edff50e02c\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998906    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-kube-api-access-xjgvj\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-kube-api-access-xjgvj\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.998934    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-envoy\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999088    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d61337-0a44-4b54-8259-45b7c4467893-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d61337-0a44-4b54-8259-45b7c4467893-istio-podinfo\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999147    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-socket\") pod \"45d61337-0a44-4b54-8259-45b7c4467893\" (UID: \"45d61337-0a44-4b54-8259-45b7c4467893\") "
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999506    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999804    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjc6\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-kube-api-access-6qjc6\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999856    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999908    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999937    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:00.999984    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.000086    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd1809-2911-4280-8828-00a2fb1569fb-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.000693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.000156    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.002244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.000969    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.003297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.003133    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-certs" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.008403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.008380    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-socket" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.016264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.016240    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-podinfo" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:18:01.016632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.016609    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istiod-ca-cert" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:01.019952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.019487    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-certs" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.019952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.019724    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d61337-0a44-4b54-8259-45b7c4467893-istiod-ca-cert" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:01.020390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.020260    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-credential-socket" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.020465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.020446    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-data" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.023508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.020701    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-envoy" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.023508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.020778    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-socket" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.023508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.020955    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-credential-socket" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.023508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.022641    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-data" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.029060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.029017    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-kube-api-access-mvtpn" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "kube-api-access-mvtpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:01.029788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.029707    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:18:01.031260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.031232    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d61337-0a44-4b54-8259-45b7c4467893-istio-podinfo" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:18:01.040881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.040851    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-kube-api-access-xjgvj" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "kube-api-access-xjgvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:01.041000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.040893    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-istio-token" pod "45d61337-0a44-4b54-8259-45b7c4467893" (UID: "45d61337-0a44-4b54-8259-45b7c4467893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:01.041070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.041019    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-token" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:01.043103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.043070    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-envoy" pod "bbeed7de-92f1-4222-b9cd-e7edff50e02c" (UID: "bbeed7de-92f1-4222-b9cd-e7edff50e02c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.103133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.103061    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-podinfo\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.103543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.103223    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-certs\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.103616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.103559    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-certs" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.103616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.103572    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-data\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.103794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.103710    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-kube-api-access-scxfd\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-kube-api-access-scxfd\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.103853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.103785    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-data" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.103853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.103843    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-token\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.104135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.104109    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-envoy\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.104205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.104195    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-credential-socket\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.104485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.104462    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-credential-socket" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.104700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.104632    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-socket\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.104976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.104953    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-socket" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.105516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.105490    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-podinfo" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:18:01.105589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.105547    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istiod-ca-cert\") pod \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\" (UID: \"bbd3322e-04e9-4d6d-98ad-e77ca340566a\") "
Nov 02 15:18:01.106209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.106186    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.106443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.106427    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjc6\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-kube-api-access-6qjc6\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.106491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.106476    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.106546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.106529    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.106585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.106572    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.106635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.106619    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.107098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.107078    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.107450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.107426    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istiod-ca-cert" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:01.107781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.107756    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-kube-api-access-scxfd" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "kube-api-access-scxfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:01.107873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.107796    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-certs\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.108058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.108031    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-credential-socket\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.108945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.108921    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd1809-2911-4280-8828-00a2fb1569fb-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.109150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.109129    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.109354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.109333    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.109492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.109466    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd1809-2911-4280-8828-00a2fb1569fb-istiod-ca-cert\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.109788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.109666    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-data\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.109883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.109805    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-envoy" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110014    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110031    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxfd\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-kube-api-access-scxfd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110041    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110050    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjgvj\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-kube-api-access-xjgvj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110059    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110068    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d61337-0a44-4b54-8259-45b7c4467893-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110076    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110083    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110092    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110100    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110724    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110740    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110753    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110768    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110781    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d61337-0a44-4b54-8259-45b7c4467893-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110792    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110803    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110815    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110827    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d61337-0a44-4b54-8259-45b7c4467893-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110839    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110852    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvtpn\" (UniqueName: \"kubernetes.io/projected/bbeed7de-92f1-4222-b9cd-e7edff50e02c-kube-api-access-mvtpn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110865    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbeed7de-92f1-4222-b9cd-e7edff50e02c-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110878    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d61337-0a44-4b54-8259-45b7c4467893-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110890    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.110903    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbeed7de-92f1-4222-b9cd-e7edff50e02c-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.111049    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-podinfo\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.112372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.112058    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-token" pod "bbd3322e-04e9-4d6d-98ad-e77ca340566a" (UID: "bbd3322e-04e9-4d6d-98ad-e77ca340566a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:01.113926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.113890    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-envoy\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.142616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.142538    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-token\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.142616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.142556    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjc6\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-kube-api-access-6qjc6\") pod \"e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") " pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.212674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.212623    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.212889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.212866    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd3322e-04e9-4d6d-98ad-e77ca340566a-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:01.326237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.326200    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:01.670504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.669762    7019 generic.go:356] "Generic (PLEG): container finished" podID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerID="5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4" exitCode=0
Nov 02 15:18:01.670504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.669831    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"
Nov 02 15:18:01.670504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.669842    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" event={"ID":"bbd3322e-04e9-4d6d-98ad-e77ca340566a","Type":"ContainerDied","Data":"5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4"}
Nov 02 15:18:01.670504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.669945    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh" event={"ID":"bbd3322e-04e9-4d6d-98ad-e77ca340566a","Type":"ContainerDied","Data":"af4c92607dded5fa11d9e88d78e4cf12907f30f17c81becf31582fe1890ab537"}
Nov 02 15:18:01.670504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.669981    7019 scope.go:122] "RemoveContainer" containerID="5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4"
Nov 02 15:18:01.673549 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:18:01.673520    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdd1809_2911_4280_8828_00a2fb1569fb.slice/crio-e6954caa8a7ddb2cb0274095afb0086e8cc6e767818f02430e93517b80e6edb3 WatchSource:0}: Error finding container e6954caa8a7ddb2cb0274095afb0086e8cc6e767818f02430e93517b80e6edb3: Status 404 returned error can't find the container with id e6954caa8a7ddb2cb0274095afb0086e8cc6e767818f02430e93517b80e6edb3
Nov 02 15:18:01.674216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.674186    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt" event={"ID":"bbeed7de-92f1-4222-b9cd-e7edff50e02c","Type":"ContainerDied","Data":"f4b827ad9b3bc7276319ee98179fbc13b47be60c0d8e766e41ef85f7c76bd458"}
Nov 02 15:18:01.674339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.674319    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"
Nov 02 15:18:01.674493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.674462    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"]
Nov 02 15:18:01.677389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.677359    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn" event={"ID":"45d61337-0a44-4b54-8259-45b7c4467893","Type":"ContainerDied","Data":"5f8879ce4f7304fa8dbb087b56a261378a8ac3a51037c552242b5d0232d81325"}
Nov 02 15:18:01.677514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.677478    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"
Nov 02 15:18:01.677595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.677554    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:01.677700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.677668    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:01.677772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.677727    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:01.699794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.699767    7019 scope.go:122] "RemoveContainer" containerID="5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4"
Nov 02 15:18:01.700428 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:01.700346    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4\": container with ID starting with 5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4 not found: ID does not exist" containerID="5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4"
Nov 02 15:18:01.700428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.700384    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4"} err="failed to get container status \"5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4\": rpc error: code = NotFound desc = could not find container \"5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4\": container with ID starting with 5d5e35f5e2073f1fd935fb9dbf63085548952341cb9a4743fb3da9d07da75cb4 not found: ID does not exist"
Nov 02 15:18:01.700428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.700409    7019 scope.go:122] "RemoveContainer" containerID="f0ad15fd25efb1c7e79b61d609e09c2923e105af4c0a0a8a5fd55a8a00774df1"
Nov 02 15:18:01.722616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.722587    7019 scope.go:122] "RemoveContainer" containerID="6ebdc2e00691930b42a59a9649ae2fb246d661804cd933f0ba0e89139572d5f1"
Nov 02 15:18:01.761660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.761617    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"]
Nov 02 15:18:01.780242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.780205    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-67xxh"]
Nov 02 15:18:01.809405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.809344    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"]
Nov 02 15:18:01.814257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.814228    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-xlrkt"]
Nov 02 15:18:01.853142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.853096    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"]
Nov 02 15:18:01.868741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.868700    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" path="/var/lib/kubelet/pods/bbd3322e-04e9-4d6d-98ad-e77ca340566a/volumes"
Nov 02 15:18:01.869527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.869501    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbeed7de-92f1-4222-b9cd-e7edff50e02c" path="/var/lib/kubelet/pods/bbeed7de-92f1-4222-b9cd-e7edff50e02c/volumes"
Nov 02 15:18:01.869978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:01.869956    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-5slfn"]
Nov 02 15:18:02.013559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.013471    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"]
Nov 02 15:18:02.014933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.014903    7019 cpu_manager.go:414] "containerMap: removing container" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy"
Nov 02 15:18:02.014933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.014932    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy"
Nov 02 15:18:02.015182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.014990    7019 cpu_manager.go:414] "containerMap: removing container" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy"
Nov 02 15:18:02.015182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.014998    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy"
Nov 02 15:18:02.015286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.015208    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d61337-0a44-4b54-8259-45b7c4467893" containerName="istio-proxy"
Nov 02 15:18:02.015286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.015229    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd3322e-04e9-4d6d-98ad-e77ca340566a" containerName="istio-proxy"
Nov 02 15:18:02.020460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.020436    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.042888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.042857    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"]
Nov 02 15:18:02.047533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.047504    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"]
Nov 02 15:18:02.052981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.052944    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.088575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.088542    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"]
Nov 02 15:18:02.129383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.128706    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.129383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.128782    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.129383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.128905    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.129383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.129001    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.133273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.129074    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.133739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.133618    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.133977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.133935    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.134185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.134144    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.134342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.134324    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx5xj\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-kube-api-access-cx5xj\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.134498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.134478    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.134640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.134617    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.134807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.134782    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwzt\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-kube-api-access-snwzt\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.134952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.134928    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.135123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.135093    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.135607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.135571    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.152020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.151976    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.152555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.152527    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.153057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.153002    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255021    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255073    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255116    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255148    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx5xj\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-kube-api-access-cx5xj\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255173    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255195    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.255245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255219    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwzt\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-kube-api-access-snwzt\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.255565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255245    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255270    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.255565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255309    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.255565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255337    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.255565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255424    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255490    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.255565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255541    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255628    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.255959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.255705    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.256075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256061    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.256135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256078    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.256135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256108    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.256244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256183    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.256244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256193    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-credential-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.256244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256223    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.257001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256786    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.257001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256934    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.257001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.256965    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istiod-ca-cert\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.257272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.257215    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-socket\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.257505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.257481    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-data\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.257608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.257525    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-certs\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.259172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.259136    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.259577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.259549    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-podinfo\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.259709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.259585    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.259842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.259822    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-envoy\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.281313 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.281242    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx5xj\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-kube-api-access-cx5xj\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.281313 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.281265    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.282372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.282345    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-token\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.282855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.282837    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwzt\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-kube-api-access-snwzt\") pod \"e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") " pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.342334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.341991    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:02.388333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.387957    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:02.610977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.610743    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:02.733454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.732182    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"]
Nov 02 15:18:02.737453 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:18:02.737401    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4a3628_bdc9_4e16_8eb8_a46dd09572b0.slice/crio-3871b1595b02a6339ec9e044a903d0718aec3116817f79cf724f8937d3692e59 WatchSource:0}: Error finding container 3871b1595b02a6339ec9e044a903d0718aec3116817f79cf724f8937d3692e59: Status 404 returned error can't find the container with id 3871b1595b02a6339ec9e044a903d0718aec3116817f79cf724f8937d3692e59
Nov 02 15:18:02.741134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.741103    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" event={"ID":"2bdd1809-2911-4280-8828-00a2fb1569fb","Type":"ContainerStarted","Data":"e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408"}
Nov 02 15:18:02.741301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.741159    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" event={"ID":"2bdd1809-2911-4280-8828-00a2fb1569fb","Type":"ContainerStarted","Data":"e6954caa8a7ddb2cb0274095afb0086e8cc6e767818f02430e93517b80e6edb3"}
Nov 02 15:18:02.741491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.741454    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.741559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.741545    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.741607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.741593    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.805291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.805197    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"]
Nov 02 15:18:02.813371 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:18:02.813324    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771bdc54_87f5_4666_a7b1_bb77ad2197f8.slice/crio-cd04b08f0bbfbb5f91f616d7ef98604db5e01787fbe6351e19b682c4a7ec56ad WatchSource:0}: Error finding container cd04b08f0bbfbb5f91f616d7ef98604db5e01787fbe6351e19b682c4a7ec56ad: Status 404 returned error can't find the container with id cd04b08f0bbfbb5f91f616d7ef98604db5e01787fbe6351e19b682c4a7ec56ad
Nov 02 15:18:02.820236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.820058    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podStartSLOduration=2.820039925 podStartE2EDuration="2.820039925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:02.814752827 +0000 UTC m=+1277.068706451" watchObservedRunningTime="2026-11-02 15:18:02.820039925 +0000 UTC m=+1277.073993550"
Nov 02 15:18:02.824915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.824878    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.825018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.824977    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:02.825078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:02.825029    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:18:03.326713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.326398    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:03.327848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.327821    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:18:03.328060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.328031    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 15:18:03.755713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.755666    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" event={"ID":"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0","Type":"ContainerStarted","Data":"4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20"}
Nov 02 15:18:03.756032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.755738    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" event={"ID":"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0","Type":"ContainerStarted","Data":"3871b1595b02a6339ec9e044a903d0718aec3116817f79cf724f8937d3692e59"}
Nov 02 15:18:03.757496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.757467    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" event={"ID":"771bdc54-87f5-4666-a7b1-bb77ad2197f8","Type":"ContainerStarted","Data":"5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd"}
Nov 02 15:18:03.757606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.757513    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" event={"ID":"771bdc54-87f5-4666-a7b1-bb77ad2197f8","Type":"ContainerStarted","Data":"cd04b08f0bbfbb5f91f616d7ef98604db5e01787fbe6351e19b682c4a7ec56ad"}
Nov 02 15:18:03.871673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.864837    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d61337-0a44-4b54-8259-45b7c4467893" path="/var/lib/kubelet/pods/45d61337-0a44-4b54-8259-45b7c4467893/volumes"
Nov 02 15:18:03.902750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.902695    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podStartSLOduration=2.90267315 podStartE2EDuration="2.90267315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:03.83030346 +0000 UTC m=+1278.084257081" watchObservedRunningTime="2026-11-02 15:18:03.90267315 +0000 UTC m=+1278.156626776"
Nov 02 15:18:03.904014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:03.903976    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podStartSLOduration=1.903966021 podStartE2EDuration="1.903966021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:03.888368059 +0000 UTC m=+1278.142321686" watchObservedRunningTime="2026-11-02 15:18:03.903966021 +0000 UTC m=+1278.157919645"
Nov 02 15:18:04.336671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.327077    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:18:04.336671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.327135    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 15:18:04.349683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.349440    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:04.349683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.349584    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 15:18:04.349683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.349628    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 15:18:04.414670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.411803    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:04.414670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.412419    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:04.414670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.412568    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 15:18:04.470592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:04.470557    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:05.153685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:05.153627    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:05.329859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:05.329775    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:18:05.329859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:05.329839    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 15:18:05.345074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:05.345031    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 15:18:05.345680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:05.345623    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 15:18:05.389172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:05.389134    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:05.389488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:05.389444    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 15:18:06.283672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:06.283265    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:06.339669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:06.329096    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:18:06.339669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:06.329158    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 15:18:06.343669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:06.343012    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 15:18:06.343669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:06.343076    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 15:18:06.404681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:06.404308    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:06.404681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:06.404370    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 15:18:07.335673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:07.333507    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:18:07.335673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:07.333566    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 15:18:07.354722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:07.346413    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 15:18:07.354722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:07.346472    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 15:18:07.394091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:07.394053    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:07.394379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:07.394352    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 15:18:08.103672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:08.096448    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:08.335098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:08.335058    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 15:18:08.335382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:08.335351    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 15:18:08.350504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:08.350468    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 15:18:08.350788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:08.350762    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 15:18:08.403275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:08.403137    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:08.403275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:08.403200    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 15:18:08.475673 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:08.462599    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:08.475673 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:08.462706    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:09.333981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:09.333946    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:09.334837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:09.334815    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:09.335515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:09.335495    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:18:09.344558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:09.344532    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 15:18:09.344636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:09.344577    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 15:18:09.409672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:09.408752    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:09.409672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:09.408828    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 15:18:10.120681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:10.120606    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:10.343763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:10.343720    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 02 15:18:10.344173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:10.343808    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 02 15:18:10.389162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:10.389062    7019 patch_prober.go:29] interesting pod/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 02 15:18:10.389345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:10.389148    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 02 15:18:11.348110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:11.348080    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:11.348579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:11.348560    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:11.349424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:11.349403    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:18:11.394406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:11.394369    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:11.394788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:11.394769    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:11.395668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:11.395633    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:18:11.795901 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:11.795855    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:11.796118 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:11.795904    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:12.050638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:12.050224    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:12.442012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:12.441971    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:13.639700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:13.639665    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:14.530828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:14.530785    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:14.713896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:14.713858    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:15.400695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:15.400657    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:16.954323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:16.954239    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:18:17.616704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:17.616605    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:18.236595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:18.236552    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:19.061339 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:19.061298    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:19.358807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:19.356512    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:19.931681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:19.931556    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:20.087512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:20.087477    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:21.990035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:21.989994    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:22.707870 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:22.707828    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:18:23.002217 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:23.002129    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:23.889358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:23.889321    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:24.209231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:24.209138    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:25.550787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:25.550748    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:25.723512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:25.723475    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:25.936667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:25.933759    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:26.115344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:26.115308    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:27.923484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:27.923443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:29.188887 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:29.188845    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:29.591542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:29.591392    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:31.194242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:31.194202    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:31.399910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:31.399869    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:31.546775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:31.546689    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:33.266353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:33.266318    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:34.338047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:34.338011    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:34.866310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:34.866268    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:35.039326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:35.036746    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:36.589603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:36.589563    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:38.054910 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:38.054871    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:38.294657 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.294602    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"]
Nov 02 15:18:38.299816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.299790    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.334717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.334681    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:38.336887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.336845    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:18:38.347738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.347062    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"]
Nov 02 15:18:38.375162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.374977    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:38.453441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.453340    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdn4t\" (UniqueName: \"kubernetes.io/projected/fea51879-97e9-417d-9975-a3c5383be80e-kube-api-access-fdn4t\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.453927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.453900    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.454283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.454262    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.556230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.556188    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.556448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.556392    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdn4t\" (UniqueName: \"kubernetes.io/projected/fea51879-97e9-417d-9975-a3c5383be80e-kube-api-access-fdn4t\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.556448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.556435    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.556898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.556876    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.558813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.558791    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.583019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.582947    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdn4t\" (UniqueName: \"kubernetes.io/projected/fea51879-97e9-417d-9975-a3c5383be80e-kube-api-access-fdn4t\") pod \"maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.650417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.650381    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:38.957575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:38.957537    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"]
Nov 02 15:18:38.958088 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:18:38.958064    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea51879_97e9_417d_9975_a3c5383be80e.slice/crio-df6e75e063995330986ac53a309bbf1c662bc27ea76d6163713a8d578059a960 WatchSource:0}: Error finding container df6e75e063995330986ac53a309bbf1c662bc27ea76d6163713a8d578059a960: Status 404 returned error can't find the container with id df6e75e063995330986ac53a309bbf1c662bc27ea76d6163713a8d578059a960
Nov 02 15:18:39.063696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:39.061127    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:39.230454 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:39.230352    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d8223_7b91_4d72_9073_bfc6afa6888c.slice/crio-conmon-9f248d0ead3367e4c8b1012a45dd24dabe3931f3cb87b7c84798c3ceaa0b28da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:18:39.344682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:39.344379    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:39.440965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:39.440912    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv" event={"ID":"fea51879-97e9-417d-9975-a3c5383be80e","Type":"ContainerStarted","Data":"df6e75e063995330986ac53a309bbf1c662bc27ea76d6163713a8d578059a960"}
Nov 02 15:18:40.450378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:40.450342    7019 generic.go:356] "Generic (PLEG): container finished" podID="fea51879-97e9-417d-9975-a3c5383be80e" containerID="082b52080ca884cc4d0ce8065abc938e5629d4af2a5503a49c3ccd9a18eb3b61" exitCode=0
Nov 02 15:18:40.450793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:40.450426    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv" event={"ID":"fea51879-97e9-417d-9975-a3c5383be80e","Type":"ContainerDied","Data":"082b52080ca884cc4d0ce8065abc938e5629d4af2a5503a49c3ccd9a18eb3b61"}
Nov 02 15:18:40.529683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:40.529567    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:40.985196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:40.985151    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:41.094473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.094440    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"]
Nov 02 15:18:41.098984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.098960    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.125202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.125163    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"]
Nov 02 15:18:41.127559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.127509    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a713aa\""
Nov 02 15:18:41.127703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.127677    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a713aa-dockercfg-d5cws\""
Nov 02 15:18:41.151988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.151951    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:41.239281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.230987    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.239281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.237746    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84q79\" (UniqueName: \"kubernetes.io/projected/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-kube-api-access-84q79\") pod \"payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.470934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.470884    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.471846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.471818    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84q79\" (UniqueName: \"kubernetes.io/projected/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-kube-api-access-84q79\") pod \"payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.474697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.474672    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.517164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.517034    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84q79\" (UniqueName: \"kubernetes.io/projected/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-kube-api-access-84q79\") pod \"payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.731361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.731323    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:41.738094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.738063    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:41.881051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.881019    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea51879-97e9-417d-9975-a3c5383be80e-kube-api-access-fdn4t\" (UniqueName: \"kubernetes.io/projected/fea51879-97e9-417d-9975-a3c5383be80e-kube-api-access-fdn4t\") pod \"fea51879-97e9-417d-9975-a3c5383be80e\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") "
Nov 02 15:18:41.881235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.881059    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-xks-ca\") pod \"fea51879-97e9-417d-9975-a3c5383be80e\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") "
Nov 02 15:18:41.881235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.881086    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-openshift-ca\") pod \"fea51879-97e9-417d-9975-a3c5383be80e\" (UID: \"fea51879-97e9-417d-9975-a3c5383be80e\") "
Nov 02 15:18:41.881535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.881495    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-openshift-ca" pod "fea51879-97e9-417d-9975-a3c5383be80e" (UID: "fea51879-97e9-417d-9975-a3c5383be80e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:41.882104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.882082    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:41.884818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.884790    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-xks-ca" pod "fea51879-97e9-417d-9975-a3c5383be80e" (UID: "fea51879-97e9-417d-9975-a3c5383be80e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:41.884818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.884807    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea51879-97e9-417d-9975-a3c5383be80e-kube-api-access-fdn4t" pod "fea51879-97e9-417d-9975-a3c5383be80e" (UID: "fea51879-97e9-417d-9975-a3c5383be80e"). InnerVolumeSpecName "kube-api-access-fdn4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:41.984385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.984346    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdn4t\" (UniqueName: \"kubernetes.io/projected/fea51879-97e9-417d-9975-a3c5383be80e-kube-api-access-fdn4t\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:41.984385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:41.984385    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fea51879-97e9-417d-9975-a3c5383be80e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:42.056378 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:18:42.056319    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a52df5_81c4_47b7_9d9a_fb632f71ceb8.slice/crio-492f6f22c2183b884ac33a62e19b24dceb5af4c50f8097971c578bb8a8c74282 WatchSource:0}: Error finding container 492f6f22c2183b884ac33a62e19b24dceb5af4c50f8097971c578bb8a8c74282: Status 404 returned error can't find the container with id 492f6f22c2183b884ac33a62e19b24dceb5af4c50f8097971c578bb8a8c74282
Nov 02 15:18:42.057069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.057042    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"]
Nov 02 15:18:42.486987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.486957    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"
Nov 02 15:18:42.487461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.486953    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv" event={"ID":"fea51879-97e9-417d-9975-a3c5383be80e","Type":"ContainerDied","Data":"df6e75e063995330986ac53a309bbf1c662bc27ea76d6163713a8d578059a960"}
Nov 02 15:18:42.487461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.487099    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6e75e063995330986ac53a309bbf1c662bc27ea76d6163713a8d578059a960"
Nov 02 15:18:42.493729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.493696    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" event={"ID":"37a52df5-81c4-47b7-9d9a-fb632f71ceb8","Type":"ContainerStarted","Data":"eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702"}
Nov 02 15:18:42.493894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.493754    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" event={"ID":"37a52df5-81c4-47b7-9d9a-fb632f71ceb8","Type":"ContainerStarted","Data":"492f6f22c2183b884ac33a62e19b24dceb5af4c50f8097971c578bb8a8c74282"}
Nov 02 15:18:42.493991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.493976    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:42.494688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.494624    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:42.554666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:42.554592    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" podStartSLOduration=1.5545720840000001 podStartE2EDuration="1.554572084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:42.55134654 +0000 UTC m=+1316.805300166" watchObservedRunningTime="2026-11-02 15:18:42.554572084 +0000 UTC m=+1316.808525709"
Nov 02 15:18:44.173677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:44.173613    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:45.827984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:45.826129    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:45.969833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:45.969791    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:47.261057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:47.261027    7019 scope.go:122] "RemoveContainer" containerID="f79e7190718c173f7f308a86caef602645cd6f0b8fba33e4bb3f32dfbc042bdd"
Nov 02 15:18:47.440758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:47.440718    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:47.595792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:47.595701    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:48.921996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:48.921960    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:49.065637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.065601    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:49.212749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.212620    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:49.354680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.354634    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"]
Nov 02 15:18:49.355436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.355416    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fea51879-97e9-417d-9975-a3c5383be80e" containerName="revoke-keys"
Nov 02 15:18:49.355436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.355436    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea51879-97e9-417d-9975-a3c5383be80e" containerName="revoke-keys"
Nov 02 15:18:49.355694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.355664    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea51879-97e9-417d-9975-a3c5383be80e" containerName="revoke-keys"
Nov 02 15:18:49.361674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.361117    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.390639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.390058    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:18:49.416094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.408229    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:18:49.517960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.490004    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.517960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.490080    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.517960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.506970    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6vkw\" (UniqueName: \"kubernetes.io/projected/b54a83c3-0a90-4813-8a26-6a873151483b-kube-api-access-k6vkw\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.678240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.678205    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"]
Nov 02 15:18:49.682721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.682690    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.692845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.692805    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.694875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.694835    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6vkw\" (UniqueName: \"kubernetes.io/projected/b54a83c3-0a90-4813-8a26-6a873151483b-kube-api-access-k6vkw\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.702302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.702272    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.703474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.703359    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.722174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.722111    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6vkw\" (UniqueName: \"kubernetes.io/projected/b54a83c3-0a90-4813-8a26-6a873151483b-kube-api-access-k6vkw\") pod \"maas-api-revoke-sub-8f043844-xj7c4\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:49.879092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:49.879058    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:50.188984 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:18:50.188938    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54a83c3_0a90_4813_8a26_6a873151483b.slice/crio-1359cf44e4b0d73af9ef7d09e6f1647b75b2604d4509e910555ffa8466d1869b WatchSource:0}: Error finding container 1359cf44e4b0d73af9ef7d09e6f1647b75b2604d4509e910555ffa8466d1869b: Status 404 returned error can't find the container with id 1359cf44e4b0d73af9ef7d09e6f1647b75b2604d4509e910555ffa8466d1869b
Nov 02 15:18:50.190478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:50.190449    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"]
Nov 02 15:18:50.710662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:50.710610    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4" event={"ID":"b54a83c3-0a90-4813-8a26-6a873151483b","Type":"ContainerStarted","Data":"3712e1581924667be4c0442a609c8bbb044a15eeecbd936effa5b0b576f39c98"}
Nov 02 15:18:50.710843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:50.710700    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4" event={"ID":"b54a83c3-0a90-4813-8a26-6a873151483b","Type":"ContainerStarted","Data":"1359cf44e4b0d73af9ef7d09e6f1647b75b2604d4509e910555ffa8466d1869b"}
Nov 02 15:18:50.757754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:50.757626    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4" podStartSLOduration=1.757610924 podStartE2EDuration="1.757610924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:50.756707724 +0000 UTC m=+1325.010661371" watchObservedRunningTime="2026-11-02 15:18:50.757610924 +0000 UTC m=+1325.011564579"
Nov 02 15:18:50.806531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:50.806492    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:51.720018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:51.719981    7019 generic.go:356] "Generic (PLEG): container finished" podID="b54a83c3-0a90-4813-8a26-6a873151483b" containerID="3712e1581924667be4c0442a609c8bbb044a15eeecbd936effa5b0b576f39c98" exitCode=0
Nov 02 15:18:51.720465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:51.720069    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4" event={"ID":"b54a83c3-0a90-4813-8a26-6a873151483b","Type":"ContainerDied","Data":"3712e1581924667be4c0442a609c8bbb044a15eeecbd936effa5b0b576f39c98"}
Nov 02 15:18:52.458080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.458044    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:52.711217 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:18:52.711004    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:18:52.899252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.899219    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:52.965896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.965803    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-openshift-ca\") pod \"b54a83c3-0a90-4813-8a26-6a873151483b\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") "
Nov 02 15:18:52.966077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.966000    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54a83c3-0a90-4813-8a26-6a873151483b-kube-api-access-k6vkw\" (UniqueName: \"kubernetes.io/projected/b54a83c3-0a90-4813-8a26-6a873151483b-kube-api-access-k6vkw\") pod \"b54a83c3-0a90-4813-8a26-6a873151483b\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") "
Nov 02 15:18:52.966254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.966221    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-openshift-ca" pod "b54a83c3-0a90-4813-8a26-6a873151483b" (UID: "b54a83c3-0a90-4813-8a26-6a873151483b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:18:52.966254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.966240    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-xks-ca\") pod \"b54a83c3-0a90-4813-8a26-6a873151483b\" (UID: \"b54a83c3-0a90-4813-8a26-6a873151483b\") "
Nov 02 15:18:52.967216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.967187    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:52.968696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.968667    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54a83c3-0a90-4813-8a26-6a873151483b-kube-api-access-k6vkw" pod "b54a83c3-0a90-4813-8a26-6a873151483b" (UID: "b54a83c3-0a90-4813-8a26-6a873151483b"). InnerVolumeSpecName "kube-api-access-k6vkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:18:52.969051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:52.969024    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-xks-ca" pod "b54a83c3-0a90-4813-8a26-6a873151483b" (UID: "b54a83c3-0a90-4813-8a26-6a873151483b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:18:53.068792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:53.068753    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b54a83c3-0a90-4813-8a26-6a873151483b-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:53.068792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:53.068785    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6vkw\" (UniqueName: \"kubernetes.io/projected/b54a83c3-0a90-4813-8a26-6a873151483b-kube-api-access-k6vkw\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:18:53.505051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:53.505020    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:18:53.736344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:53.736314    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"
Nov 02 15:18:53.736344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:53.736331    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4" event={"ID":"b54a83c3-0a90-4813-8a26-6a873151483b","Type":"ContainerDied","Data":"1359cf44e4b0d73af9ef7d09e6f1647b75b2604d4509e910555ffa8466d1869b"}
Nov 02 15:18:53.736559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:53.736374    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1359cf44e4b0d73af9ef7d09e6f1647b75b2604d4509e910555ffa8466d1869b"
Nov 02 15:18:54.148151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:54.148109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:54.295095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:54.294006    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:55.523085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.523045    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"]
Nov 02 15:18:55.523528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.523278    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy" containerID="cri-o://ed742df789b3623a2cc88b1b096ceb5db7277db2091d09d13cee48e5346b6164" gracePeriod=30
Nov 02 15:18:55.749092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.749039    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"]
Nov 02 15:18:55.749339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.749283    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy" containerID="cri-o://874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7" gracePeriod=30
Nov 02 15:18:55.754083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.754056    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"]
Nov 02 15:18:55.754330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.754285    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy" containerID="cri-o://aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e" gracePeriod=30
Nov 02 15:18:55.754557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.754533    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:55.759150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.759121    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"]
Nov 02 15:18:55.759357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.759335    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy" containerID="cri-o://7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd" gracePeriod=30
Nov 02 15:18:55.764185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.763262    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"]
Nov 02 15:18:55.764185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.763437    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy" containerID="cri-o://5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab" gracePeriod=30
Nov 02 15:18:55.924682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.923589    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:55.968671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.967755    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"]
Nov 02 15:18:55.974669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.970566    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy" containerID="cri-o://643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196" gracePeriod=30
Nov 02 15:18:55.974669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.973901    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"]
Nov 02 15:18:55.974669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:55.974093    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy" containerID="cri-o://2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5" gracePeriod=30
Nov 02 15:18:57.804431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:57.804388    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:59.413611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:59.413507    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:18:59.846489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:18:59.846391    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:00.169009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:00.168884    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:00.750342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:00.750302    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:19:00.902207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:00.901692    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" event={"ID":"da2f7d2d-c100-4f10-bc92-17e34f8e8527","Type":"ContainerDied","Data":"ed742df789b3623a2cc88b1b096ceb5db7277db2091d09d13cee48e5346b6164"}
Nov 02 15:19:00.902207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:00.901778    7019 generic.go:356] "Generic (PLEG): container finished" podID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerID="ed742df789b3623a2cc88b1b096ceb5db7277db2091d09d13cee48e5346b6164" exitCode=0
Nov 02 15:19:01.251263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.251235    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:19:01.280353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.280318    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:01.309216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.309128    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:19:01.397419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.397363    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-certs\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.397727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.397691    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-credential-socket\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.397924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.397908    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-podinfo\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.398159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.398141    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-kube-api-access-rlv87\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-kube-api-access-rlv87\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.398361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.398344    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-data\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.398611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.398597    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-envoy\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.398786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.398771    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-envoy\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.398952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.398938    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed657d0-b5a5-4095-bc45-9d6db59c762a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed657d0-b5a5-4095-bc45-9d6db59c762a-istiod-ca-cert\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.399118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.399103    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-credential-socket\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.399338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.399323    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-podinfo\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.399566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.399548    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-certs\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.399802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.399786    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-data\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.400005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.399967    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-kube-api-access-77cbz\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-kube-api-access-77cbz\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.400246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.400207    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-token\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-token\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.400495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.400476    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-socket\") pod \"aed657d0-b5a5-4095-bc45-9d6db59c762a\" (UID: \"aed657d0-b5a5-4095-bc45-9d6db59c762a\") "
Nov 02 15:19:01.400795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.400766    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-token\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-token\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.400950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.400931    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istiod-ca-cert\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.401126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.401112    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-socket\") pod \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\" (UID: \"da2f7d2d-c100-4f10-bc92-17e34f8e8527\") "
Nov 02 15:19:01.402506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.402483    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-socket" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.404539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.403957    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-certs" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.404539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.404145    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-certs" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.404539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.404304    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-data" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.427708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.426203    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-envoy" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.436683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.436450    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-credential-socket" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.444640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.436965    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed657d0-b5a5-4095-bc45-9d6db59c762a-istiod-ca-cert" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.444640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.437440    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-data" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.444640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.440991    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-socket" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.444640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.441310    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istiod-ca-cert" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.444640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.442712    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-token" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.444640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.442749    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-kube-api-access-rlv87" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "kube-api-access-rlv87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.445075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.444870    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-token" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.445075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.445005    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-credential-socket" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.447300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.447262    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-podinfo" pod "aed657d0-b5a5-4095-bc45-9d6db59c762a" (UID: "aed657d0-b5a5-4095-bc45-9d6db59c762a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.451360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.448781    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-kube-api-access-77cbz" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "kube-api-access-77cbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.471359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.471299    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-podinfo" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.471519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.471360    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-envoy" pod "da2f7d2d-c100-4f10-bc92-17e34f8e8527" (UID: "da2f7d2d-c100-4f10-bc92-17e34f8e8527"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.509570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.509536    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed657d0-b5a5-4095-bc45-9d6db59c762a-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.509799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.509785    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.509965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.509948    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510084    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510211    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510333    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77cbz\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-kube-api-access-77cbz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510448    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510559    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510688    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510785    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.510910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510897    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.511011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.510998    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.511115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.511104    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.511217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.511206    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.511324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.511313    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlv87\" (UniqueName: \"kubernetes.io/projected/aed657d0-b5a5-4095-bc45-9d6db59c762a-kube-api-access-rlv87\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.511420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.511410    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.511554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.511543    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed657d0-b5a5-4095-bc45-9d6db59c762a-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.511640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.511631    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2f7d2d-c100-4f10-bc92-17e34f8e8527-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.522580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.521958    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:19:01.527288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.527260    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:19:01.612906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.612874    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-certs\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.612955    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-envoy\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.612986    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-token\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-token\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613038    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a555b59-c7fc-495e-ab24-1365829e3293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a555b59-c7fc-495e-ab24-1365829e3293-istiod-ca-cert\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613071    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-credential-socket\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613150    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a555b59-c7fc-495e-ab24-1365829e3293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a555b59-c7fc-495e-ab24-1365829e3293-istio-podinfo\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613194    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-socket\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613241    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-podinfo\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613316    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-credential-socket\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613355    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-data\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613411    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-kube-api-access-x6ljt\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-kube-api-access-x6ljt\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613458    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-data\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613484    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-envoy\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613659    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47728e82-309b-40f1-98d4-9cf1955fe7ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47728e82-309b-40f1-98d4-9cf1955fe7ef-istiod-ca-cert\") pod \"47728e82-309b-40f1-98d4-9cf1955fe7ef\" (UID: \"47728e82-309b-40f1-98d4-9cf1955fe7ef\") "
Nov 02 15:19:01.613762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613695    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-socket\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613766    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-certs\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613828    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-kube-api-access-rtmkg\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-kube-api-access-rtmkg\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.613974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.613908    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-istio-token\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-istio-token\") pod \"3a555b59-c7fc-495e-ab24-1365829e3293\" (UID: \"3a555b59-c7fc-495e-ab24-1365829e3293\") "
Nov 02 15:19:01.615135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.614981    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-credential-socket" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.615291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.615270    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-certs" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.615474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.615441    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-credential-socket" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.615556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.615482    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-data" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.615556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.615519    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-socket" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.615743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.615725    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47728e82-309b-40f1-98d4-9cf1955fe7ef-istiod-ca-cert" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.616352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.616085    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-socket" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.616352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.616103    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a555b59-c7fc-495e-ab24-1365829e3293-istiod-ca-cert" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.616352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.616139    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-certs" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.617153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.617127    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-data" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.619704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.619641    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-token" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.619921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.619857    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-kube-api-access-rtmkg" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "kube-api-access-rtmkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.619921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.619880    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-istio-token" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.619921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.619884    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a555b59-c7fc-495e-ab24-1365829e3293-istio-podinfo" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.619921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.619892    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-kube-api-access-x6ljt" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "kube-api-access-x6ljt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.620484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.620460    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-envoy" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.620549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.620473    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-podinfo" pod "47728e82-309b-40f1-98d4-9cf1955fe7ef" (UID: "47728e82-309b-40f1-98d4-9cf1955fe7ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.621451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.621425    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-envoy" pod "3a555b59-c7fc-495e-ab24-1365829e3293" (UID: "3a555b59-c7fc-495e-ab24-1365829e3293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.621522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.621470    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:19:01.665469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.665332    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:19:01.669840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.669781    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:19:01.715729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.715702    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-envoy\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.715889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.715769    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-certs\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.715889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.715809    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-socket\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.715889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.715870    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istiod-ca-cert\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.715991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.715954    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-data\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.716028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716015    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-token\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-token\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.716059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716036    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-certs" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.716096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716064    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-kube-api-access-zr2ks\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-kube-api-access-zr2ks\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.716195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716174    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-socket" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.716303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716281    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-data" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.716370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716332    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-podinfo\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.716428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716390    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-credential-socket\") pod \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\" (UID: \"fcb5c54c-78b6-44b9-96a3-84b99d088b3c\") "
Nov 02 15:19:01.716428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.716403    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istiod-ca-cert" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.717360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717333    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6ljt\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-kube-api-access-x6ljt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717363    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717377    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717389    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47728e82-309b-40f1-98d4-9cf1955fe7ef-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717403    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717416    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717429    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717444    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmkg\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-kube-api-access-rtmkg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717474    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717489    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a555b59-c7fc-495e-ab24-1365829e3293-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717503    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717515    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717528    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.717539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717540    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717555    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717598    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a555b59-c7fc-495e-ab24-1365829e3293-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717616    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717631    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a555b59-c7fc-495e-ab24-1365829e3293-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717692    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47728e82-309b-40f1-98d4-9cf1955fe7ef-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717707    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47728e82-309b-40f1-98d4-9cf1955fe7ef-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717722    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717736    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a555b59-c7fc-495e-ab24-1365829e3293-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.718197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.717925    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-credential-socket" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.718710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.718685    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-podinfo" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.718801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.718778    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-token" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.718856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.718834    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-envoy" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.719933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.719909    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-kube-api-access-zr2ks" pod "fcb5c54c-78b6-44b9-96a3-84b99d088b3c" (UID: "fcb5c54c-78b6-44b9-96a3-84b99d088b3c"). InnerVolumeSpecName "kube-api-access-zr2ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.818851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.818817    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-data\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.818882    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-certs\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.818955    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893f1c33-0434-4804-ab42-1d27c60bb1af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893f1c33-0434-4804-ab42-1d27c60bb1af-istiod-ca-cert\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.818989    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43a68852-928e-47c0-8213-dbf2019f33b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a68852-928e-47c0-8213-dbf2019f33b9-istio-podinfo\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819062    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-envoy\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819096    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-socket\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819120    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-istio-token\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-istio-token\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819141    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a68852-928e-47c0-8213-dbf2019f33b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a68852-928e-47c0-8213-dbf2019f33b9-istiod-ca-cert\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819147    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-data" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819168    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-kube-api-access-rx2fs\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-kube-api-access-rx2fs\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819228    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-data\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819261    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-kube-api-access-frgqk\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-kube-api-access-frgqk\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819304    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-envoy\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.819363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819340    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-podinfo\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.820049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819386    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-credential-socket\") pod \"43a68852-928e-47c0-8213-dbf2019f33b9\" (UID: \"43a68852-928e-47c0-8213-dbf2019f33b9\") "
Nov 02 15:19:01.820049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819410    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-credential-socket\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.820049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819465    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-certs\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.820049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819497    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-socket\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.820049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.819618    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-token\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-token\") pod \"893f1c33-0434-4804-ab42-1d27c60bb1af\" (UID: \"893f1c33-0434-4804-ab42-1d27c60bb1af\") "
Nov 02 15:19:01.820576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.820462    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.820576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.820490    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.820576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.820508    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2ks\" (UniqueName: \"kubernetes.io/projected/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-kube-api-access-zr2ks\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.820576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.820522    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.820576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.820535    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.820576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.820548    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb5c54c-78b6-44b9-96a3-84b99d088b3c-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.820986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.820792    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-socket" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.821483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.821196    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a68852-928e-47c0-8213-dbf2019f33b9-istiod-ca-cert" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.822379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.822291    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-certs" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.822616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.822593    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-credential-socket" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.822735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.822602    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893f1c33-0434-4804-ab42-1d27c60bb1af-istiod-ca-cert" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:01.823311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.823185    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-token" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.823493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.823465    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-credential-socket" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.823687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.823671    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-socket" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.823880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.823857    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-certs" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.824109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.823862    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-data" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.824228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.823919    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-envoy" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.824360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.824180    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-kube-api-access-rx2fs" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "kube-api-access-rx2fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.825010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.824967    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-istio-token" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.825546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.825510    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-envoy" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:01.826240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.826205    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-podinfo" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.826240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.826218    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43a68852-928e-47c0-8213-dbf2019f33b9-istio-podinfo" pod "43a68852-928e-47c0-8213-dbf2019f33b9" (UID: "43a68852-928e-47c0-8213-dbf2019f33b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:01.830864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.830836    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-kube-api-access-frgqk" pod "893f1c33-0434-4804-ab42-1d27c60bb1af" (UID: "893f1c33-0434-4804-ab42-1d27c60bb1af"). InnerVolumeSpecName "kube-api-access-frgqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:01.915614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.915517    7019 generic.go:356] "Generic (PLEG): container finished" podID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerID="aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e" exitCode=0
Nov 02 15:19:01.915614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.915573    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"
Nov 02 15:19:01.915840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.915624    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" event={"ID":"47728e82-309b-40f1-98d4-9cf1955fe7ef","Type":"ContainerDied","Data":"aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e"}
Nov 02 15:19:01.915840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.915754    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf" event={"ID":"47728e82-309b-40f1-98d4-9cf1955fe7ef","Type":"ContainerDied","Data":"a1ced69943b1fee73d903fc186cadb142f8207d3602f95f1469e7d6155724203"}
Nov 02 15:19:01.915840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.915791    7019 scope.go:122] "RemoveContainer" containerID="aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e"
Nov 02 15:19:01.917986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.917957    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v" event={"ID":"da2f7d2d-c100-4f10-bc92-17e34f8e8527","Type":"ContainerDied","Data":"54035108c3aa456995ef655d6c41da8d8515c9584d1ca605cf495b7ad0491fca"}
Nov 02 15:19:01.918104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.918021    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"
Nov 02 15:19:01.920034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.920011    7019 generic.go:356] "Generic (PLEG): container finished" podID="3a555b59-c7fc-495e-ab24-1365829e3293" containerID="874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7" exitCode=0
Nov 02 15:19:01.920146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.920134    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"
Nov 02 15:19:01.920552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.920528    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" event={"ID":"3a555b59-c7fc-495e-ab24-1365829e3293","Type":"ContainerDied","Data":"874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7"}
Nov 02 15:19:01.920664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.920580    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl" event={"ID":"3a555b59-c7fc-495e-ab24-1365829e3293","Type":"ContainerDied","Data":"edeb38912a293192e47ca7cf6e739a0c69be7af6113f9ced4c7c040d69eca3d4"}
Nov 02 15:19:01.921909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921889    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/893f1c33-0434-4804-ab42-1d27c60bb1af-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921912    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a68852-928e-47c0-8213-dbf2019f33b9-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921924    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921935    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921946    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921959    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a68852-928e-47c0-8213-dbf2019f33b9-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921972    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2fs\" (UniqueName: \"kubernetes.io/projected/43a68852-928e-47c0-8213-dbf2019f33b9-kube-api-access-rx2fs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921981    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921990    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frgqk\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-kube-api-access-frgqk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.921998    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922008    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922018    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922029    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922041    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922055    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/893f1c33-0434-4804-ab42-1d27c60bb1af-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922069    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/893f1c33-0434-4804-ab42-1d27c60bb1af-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922081    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a68852-928e-47c0-8213-dbf2019f33b9-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922240    7019 generic.go:356] "Generic (PLEG): container finished" podID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerID="643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196" exitCode=0
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922290    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922330    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" event={"ID":"893f1c33-0434-4804-ab42-1d27c60bb1af","Type":"ContainerDied","Data":"643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196"}
Nov 02 15:19:01.922434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.922382    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69" event={"ID":"893f1c33-0434-4804-ab42-1d27c60bb1af","Type":"ContainerDied","Data":"99f21746a467a912fc716ad5e3a8bd9e2bc535ca709fc758013dd1a85fc38f41"}
Nov 02 15:19:01.924196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.924174    7019 generic.go:356] "Generic (PLEG): container finished" podID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerID="7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd" exitCode=0
Nov 02 15:19:01.924306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.924236    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"
Nov 02 15:19:01.924306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.924270    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" event={"ID":"aed657d0-b5a5-4095-bc45-9d6db59c762a","Type":"ContainerDied","Data":"7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd"}
Nov 02 15:19:01.924429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.924337    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l" event={"ID":"aed657d0-b5a5-4095-bc45-9d6db59c762a","Type":"ContainerDied","Data":"26c9a2da82d246d4b83633fdb7ec3493caab97752c172b85149359b7f46720d4"}
Nov 02 15:19:01.926040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.926017    7019 generic.go:356] "Generic (PLEG): container finished" podID="43a68852-928e-47c0-8213-dbf2019f33b9" containerID="2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5" exitCode=0
Nov 02 15:19:01.926150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.926099    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" event={"ID":"43a68852-928e-47c0-8213-dbf2019f33b9","Type":"ContainerDied","Data":"2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5"}
Nov 02 15:19:01.926150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.926137    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq" event={"ID":"43a68852-928e-47c0-8213-dbf2019f33b9","Type":"ContainerDied","Data":"4d73eab5d98260c7270340e41e7bc82ea0a6ef2288095c67e461137f0806bfa7"}
Nov 02 15:19:01.926254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.926203    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"
Nov 02 15:19:01.928220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.928200    7019 generic.go:356] "Generic (PLEG): container finished" podID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerID="5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab" exitCode=0
Nov 02 15:19:01.928321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.928279    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" event={"ID":"fcb5c54c-78b6-44b9-96a3-84b99d088b3c","Type":"ContainerDied","Data":"5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab"}
Nov 02 15:19:01.928387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.928320    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw" event={"ID":"fcb5c54c-78b6-44b9-96a3-84b99d088b3c","Type":"ContainerDied","Data":"a24d441d682e2599dc3feeff8ae36b18ec71d3db8301937f400284b3a74109c6"}
Nov 02 15:19:01.928456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.928384    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"
Nov 02 15:19:01.930544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.930523    7019 scope.go:122] "RemoveContainer" containerID="aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e"
Nov 02 15:19:01.930949 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:01.930919    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e\": container with ID starting with aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e not found: ID does not exist" containerID="aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e"
Nov 02 15:19:01.931030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.930964    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e"} err="failed to get container status \"aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e\": rpc error: code = NotFound desc = could not find container \"aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e\": container with ID starting with aacf870d3b59caebd6c302d62152e17370a3b30a84e31946a40b09d6334e6f5e not found: ID does not exist"
Nov 02 15:19:01.931030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.930988    7019 scope.go:122] "RemoveContainer" containerID="ed742df789b3623a2cc88b1b096ceb5db7277db2091d09d13cee48e5346b6164"
Nov 02 15:19:01.942277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.942257    7019 scope.go:122] "RemoveContainer" containerID="874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7"
Nov 02 15:19:01.952684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.952667    7019 scope.go:122] "RemoveContainer" containerID="874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7"
Nov 02 15:19:01.952956 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:01.952938    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7\": container with ID starting with 874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7 not found: ID does not exist" containerID="874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7"
Nov 02 15:19:01.953006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.952965    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7"} err="failed to get container status \"874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7\": rpc error: code = NotFound desc = could not find container \"874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7\": container with ID starting with 874b60284b693b22f5cb8fe46d0678629e21fbb53e075b824f78ea9ad8a7a9c7 not found: ID does not exist"
Nov 02 15:19:01.953006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.952984    7019 scope.go:122] "RemoveContainer" containerID="643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196"
Nov 02 15:19:01.962455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.962436    7019 scope.go:122] "RemoveContainer" containerID="643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196"
Nov 02 15:19:01.962765 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:01.962740    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196\": container with ID starting with 643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196 not found: ID does not exist" containerID="643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196"
Nov 02 15:19:01.962846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.962771    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196"} err="failed to get container status \"643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196\": rpc error: code = NotFound desc = could not find container \"643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196\": container with ID starting with 643b0baa55f2343287d749fdfd31ff5d056fe35d4691af8b5436d7a34582f196 not found: ID does not exist"
Nov 02 15:19:01.962846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.962790    7019 scope.go:122] "RemoveContainer" containerID="7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd"
Nov 02 15:19:01.972879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.972859    7019 scope.go:122] "RemoveContainer" containerID="7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd"
Nov 02 15:19:01.973201 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:01.973182    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd\": container with ID starting with 7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd not found: ID does not exist" containerID="7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd"
Nov 02 15:19:01.973243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.973211    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd"} err="failed to get container status \"7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd\": rpc error: code = NotFound desc = could not find container \"7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd\": container with ID starting with 7db1dd978bd479a7aabc4993da47705d87287e86a4979d3f9628d42febe250fd not found: ID does not exist"
Nov 02 15:19:01.973243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.973230    7019 scope.go:122] "RemoveContainer" containerID="2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5"
Nov 02 15:19:01.983461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.983429    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"]
Nov 02 15:19:01.987964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.987942    7019 scope.go:122] "RemoveContainer" containerID="2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5"
Nov 02 15:19:01.988522 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:01.988492    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5\": container with ID starting with 2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5 not found: ID does not exist" containerID="2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5"
Nov 02 15:19:01.988695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.988661    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5"} err="failed to get container status \"2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5\": rpc error: code = NotFound desc = could not find container \"2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5\": container with ID starting with 2a29fea78406ce914fd26d7df220996e098b1bff656bda00cf292118958738d5 not found: ID does not exist"
Nov 02 15:19:01.988828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:01.988811    7019 scope.go:122] "RemoveContainer" containerID="5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab"
Nov 02 15:19:02.003295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.003243    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-hww69"]
Nov 02 15:19:02.063579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.063505    7019 scope.go:122] "RemoveContainer" containerID="5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab"
Nov 02 15:19:02.075131 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:02.065114    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab\": container with ID starting with 5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab not found: ID does not exist" containerID="5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab"
Nov 02 15:19:02.075131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.065160    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab"} err="failed to get container status \"5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab\": rpc error: code = NotFound desc = could not find container \"5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab\": container with ID starting with 5456b993347d06aabfb97f53b2b2480303dd4ca25e6f12800c6244fef7c273ab not found: ID does not exist"
Nov 02 15:19:02.075131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.065208    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"]
Nov 02 15:19:02.075131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.069402    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wbdsf"]
Nov 02 15:19:02.111756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.111697    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"]
Nov 02 15:19:02.119348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.119316    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-sh74l"]
Nov 02 15:19:02.178807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.178709    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"]
Nov 02 15:19:02.202028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.200352    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-zvb9v"]
Nov 02 15:19:02.227991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.220693    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"]
Nov 02 15:19:02.235019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.234978    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-rtmsq"]
Nov 02 15:19:02.241105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.241067    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"]
Nov 02 15:19:02.246448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.246255    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-wcgjw"]
Nov 02 15:19:02.271923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.271890    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"]
Nov 02 15:19:02.276958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:02.276926    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-1cc1b2-openshift-default-5f9d97bc9-ffkwl"]
Nov 02 15:19:03.200210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.200176    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:03.865952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.865917    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" path="/var/lib/kubelet/pods/3a555b59-c7fc-495e-ab24-1365829e3293/volumes"
Nov 02 15:19:03.866448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.866432    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" path="/var/lib/kubelet/pods/43a68852-928e-47c0-8213-dbf2019f33b9/volumes"
Nov 02 15:19:03.866900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.866885    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" path="/var/lib/kubelet/pods/47728e82-309b-40f1-98d4-9cf1955fe7ef/volumes"
Nov 02 15:19:03.867712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.867693    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" path="/var/lib/kubelet/pods/893f1c33-0434-4804-ab42-1d27c60bb1af/volumes"
Nov 02 15:19:03.868203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.868187    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" path="/var/lib/kubelet/pods/aed657d0-b5a5-4095-bc45-9d6db59c762a/volumes"
Nov 02 15:19:03.868668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.868638    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" path="/var/lib/kubelet/pods/da2f7d2d-c100-4f10-bc92-17e34f8e8527/volumes"
Nov 02 15:19:03.869388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:03.869372    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" path="/var/lib/kubelet/pods/fcb5c54c-78b6-44b9-96a3-84b99d088b3c/volumes"
Nov 02 15:19:04.728992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:04.728958    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:05.399439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:05.399404    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:06.580756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:06.580007    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:07.198453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:07.198418    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:08.361136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:08.361090    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:09.917896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:09.917854    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:11.026293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.026244    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:11.399038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.399000    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"]
Nov 02 15:19:11.399997 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.399974    7019 cpu_manager.go:414] "containerMap: removing container" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy"
Nov 02 15:19:11.400073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.399998    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy"
Nov 02 15:19:11.400073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400020    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b54a83c3-0a90-4813-8a26-6a873151483b" containerName="revoke-keys"
Nov 02 15:19:11.400073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400027    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54a83c3-0a90-4813-8a26-6a873151483b" containerName="revoke-keys"
Nov 02 15:19:11.400073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400044    7019 cpu_manager.go:414] "containerMap: removing container" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy"
Nov 02 15:19:11.400073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400051    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy"
Nov 02 15:19:11.400073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400073    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400080    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400097    7019 cpu_manager.go:414] "containerMap: removing container" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400104    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400125    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400132    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400160    7019 cpu_manager.go:414] "containerMap: removing container" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400170    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400193    7019 cpu_manager.go:414] "containerMap: removing container" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy"
Nov 02 15:19:11.400262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400199    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400349    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a68852-928e-47c0-8213-dbf2019f33b9" containerName="istio-proxy"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400361    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="893f1c33-0434-4804-ab42-1d27c60bb1af" containerName="istio-proxy"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400373    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2f7d2d-c100-4f10-bc92-17e34f8e8527" containerName="istio-proxy"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400382    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54a83c3-0a90-4813-8a26-6a873151483b" containerName="revoke-keys"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400391    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="47728e82-309b-40f1-98d4-9cf1955fe7ef" containerName="istio-proxy"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400401    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb5c54c-78b6-44b9-96a3-84b99d088b3c" containerName="istio-proxy"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400410    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed657d0-b5a5-4095-bc45-9d6db59c762a" containerName="istio-proxy"
Nov 02 15:19:11.400523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.400421    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a555b59-c7fc-495e-ab24-1365829e3293" containerName="istio-proxy"
Nov 02 15:19:11.405190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.405164    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.423727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.423697    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"]
Nov 02 15:19:11.423882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.423728    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a713aa-serving-cert\""
Nov 02 15:19:11.499868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.499836    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-maas-api-tls\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.500082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.499981    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfph\" (UniqueName: \"kubernetes.io/projected/d5302cfd-d1f6-4310-ad41-c1cce3762503-kube-api-access-csfph\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.500082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.500033    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-metrics-certs\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.500082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.500071    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5302cfd-d1f6-4310-ad41-c1cce3762503-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.512190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.512154    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:11.609310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.603516    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfph\" (UniqueName: \"kubernetes.io/projected/d5302cfd-d1f6-4310-ad41-c1cce3762503-kube-api-access-csfph\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.609310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.603575    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-metrics-certs\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.609310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.603633    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5302cfd-d1f6-4310-ad41-c1cce3762503-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.609310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.603897    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-maas-api-tls\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.609310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.605142    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5302cfd-d1f6-4310-ad41-c1cce3762503-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.630139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.628802    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-maas-api-tls\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.630334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.630212    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-metrics-certs\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.633972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.633471    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfph\" (UniqueName: \"kubernetes.io/projected/d5302cfd-d1f6-4310-ad41-c1cce3762503-kube-api-access-csfph\") pod \"maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.724399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.724308    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:11.733029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:11.733000    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:12.044599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.044568    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"]
Nov 02 15:19:12.044599 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:12.044582    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5302cfd_d1f6_4310_ad41_c1cce3762503.slice/crio-a8775ee911d624b5b43afed1e3e772a8513037031f04b820672e1d8b235f716e WatchSource:0}: Error finding container a8775ee911d624b5b43afed1e3e772a8513037031f04b820672e1d8b235f716e: Status 404 returned error can't find the container with id a8775ee911d624b5b43afed1e3e772a8513037031f04b820672e1d8b235f716e
Nov 02 15:19:12.046410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.046394    7019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:19:12.618672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.609723    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"]
Nov 02 15:19:12.645018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.644986    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"]
Nov 02 15:19:12.645254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.645097    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.662890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.662865    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:19:12.676006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.675977    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:19:12.776427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.776332    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwxdt\" (UniqueName: \"kubernetes.io/projected/929f4b30-e688-40f1-a9a1-3c993623cd4f-kube-api-access-rwxdt\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.776693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.776670    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.776786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.776767    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.878024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.877984    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.878202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.878047    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.878202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.878180    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwxdt\" (UniqueName: \"kubernetes.io/projected/929f4b30-e688-40f1-a9a1-3c993623cd4f-kube-api-access-rwxdt\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.878800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.878772    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.880622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.880594    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.914921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.914884    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwxdt\" (UniqueName: \"kubernetes.io/projected/929f4b30-e688-40f1-a9a1-3c993623cd4f-kube-api-access-rwxdt\") pod \"maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:12.967297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:12.967167    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:13.041203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:13.041113    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc" event={"ID":"d5302cfd-d1f6-4310-ad41-c1cce3762503","Type":"ContainerStarted","Data":"71a5bf4ac326bc498ed7ce1b5a9f5ec74c8b18cea2d2c3188f270101b967ac4f"}
Nov 02 15:19:13.041203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:13.041186    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc" event={"ID":"d5302cfd-d1f6-4310-ad41-c1cce3762503","Type":"ContainerStarted","Data":"a8775ee911d624b5b43afed1e3e772a8513037031f04b820672e1d8b235f716e"}
Nov 02 15:19:13.042350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:13.042323    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:13.053812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:13.053786    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:13.408667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:13.408420    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc" podStartSLOduration=2.025937844 podStartE2EDuration="2.40839739s" totalImagesPullingTime="382.459546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:13.108565611 +0000 UTC m=+1347.362519236" watchObservedRunningTime="2026-11-02 15:19:13.40839739 +0000 UTC m=+1347.662351015"
Nov 02 15:19:13.413235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:13.412356    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"]
Nov 02 15:19:13.950165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:13.950066    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:14.050503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:14.050467    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6" event={"ID":"929f4b30-e688-40f1-a9a1-3c993623cd4f","Type":"ContainerStarted","Data":"d24f85dff34be35081304688b413b82b5fbb0a5349c7dbe95e106048f66a0605"}
Nov 02 15:19:14.072318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:14.072283    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"]
Nov 02 15:19:14.086537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:14.086507    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c7afd1a6-4jnwv"]
Nov 02 15:19:14.124091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:14.124061    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:14.271002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:14.270911    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:14.592318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:14.592231    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:15.058976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:15.058938    7019 generic.go:356] "Generic (PLEG): container finished" podID="929f4b30-e688-40f1-a9a1-3c993623cd4f" containerID="950aad49c6f509dd7ac0d5e05f8e4208d0ccbc68bff1c93fb82ab0c6b650ae42" exitCode=0
Nov 02 15:19:15.059165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:15.058980    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6" event={"ID":"929f4b30-e688-40f1-a9a1-3c993623cd4f","Type":"ContainerDied","Data":"950aad49c6f509dd7ac0d5e05f8e4208d0ccbc68bff1c93fb82ab0c6b650ae42"}
Nov 02 15:19:15.865225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:15.865182    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea51879-97e9-417d-9975-a3c5383be80e" path="/var/lib/kubelet/pods/fea51879-97e9-417d-9975-a3c5383be80e/volumes"
Nov 02 15:19:16.039780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.039069    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:16.255905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.255881    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:16.317720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.317687    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-xks-ca\") pod \"929f4b30-e688-40f1-a9a1-3c993623cd4f\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") "
Nov 02 15:19:16.317931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.317916    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-openshift-ca\") pod \"929f4b30-e688-40f1-a9a1-3c993623cd4f\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") "
Nov 02 15:19:16.318020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.317955    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929f4b30-e688-40f1-a9a1-3c993623cd4f-kube-api-access-rwxdt\" (UniqueName: \"kubernetes.io/projected/929f4b30-e688-40f1-a9a1-3c993623cd4f-kube-api-access-rwxdt\") pod \"929f4b30-e688-40f1-a9a1-3c993623cd4f\" (UID: \"929f4b30-e688-40f1-a9a1-3c993623cd4f\") "
Nov 02 15:19:16.318344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.318311    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-openshift-ca" pod "929f4b30-e688-40f1-a9a1-3c993623cd4f" (UID: "929f4b30-e688-40f1-a9a1-3c993623cd4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:16.319929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.319908    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-xks-ca" pod "929f4b30-e688-40f1-a9a1-3c993623cd4f" (UID: "929f4b30-e688-40f1-a9a1-3c993623cd4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:16.320078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.320056    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929f4b30-e688-40f1-a9a1-3c993623cd4f-kube-api-access-rwxdt" pod "929f4b30-e688-40f1-a9a1-3c993623cd4f" (UID: "929f4b30-e688-40f1-a9a1-3c993623cd4f"). InnerVolumeSpecName "kube-api-access-rwxdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:16.419677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.419549    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:16.419677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.419583    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwxdt\" (UniqueName: \"kubernetes.io/projected/929f4b30-e688-40f1-a9a1-3c993623cd4f-kube-api-access-rwxdt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:16.419677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:16.419597    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/929f4b30-e688-40f1-a9a1-3c993623cd4f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:17.078152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:17.078124    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"
Nov 02 15:19:17.078663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:17.078119    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6" event={"ID":"929f4b30-e688-40f1-a9a1-3c993623cd4f","Type":"ContainerDied","Data":"d24f85dff34be35081304688b413b82b5fbb0a5349c7dbe95e106048f66a0605"}
Nov 02 15:19:17.078663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:17.078253    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d24f85dff34be35081304688b413b82b5fbb0a5349c7dbe95e106048f66a0605"
Nov 02 15:19:18.100497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.100443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:18.458964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.458933    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6900-38bc00"]
Nov 02 15:19:18.459608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.459594    7019 cpu_manager.go:414] "containerMap: removing container" podUID="929f4b30-e688-40f1-a9a1-3c993623cd4f" containerName="revoke-keys"
Nov 02 15:19:18.459676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.459609    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929f4b30-e688-40f1-a9a1-3c993623cd4f" containerName="revoke-keys"
Nov 02 15:19:18.459737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.459727    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="929f4b30-e688-40f1-a9a1-3c993623cd4f" containerName="revoke-keys"
Nov 02 15:19:18.463321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.463306    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:18.480332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.480302    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcqh6\""
Nov 02 15:19:18.482702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.482676    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6900-38bc00"]
Nov 02 15:19:18.552174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.552139    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258b4\" (UniqueName: \"kubernetes.io/projected/e8b62c5c-8046-482a-a6d8-53f957838f39-kube-api-access-258b4\") pod \"test-curl-6900-38bc00\" (UID: \"e8b62c5c-8046-482a-a6d8-53f957838f39\") " pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:18.654089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.654051    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258b4\" (UniqueName: \"kubernetes.io/projected/e8b62c5c-8046-482a-a6d8-53f957838f39-kube-api-access-258b4\") pod \"test-curl-6900-38bc00\" (UID: \"e8b62c5c-8046-482a-a6d8-53f957838f39\") " pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:18.678992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.678961    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-258b4\" (UniqueName: \"kubernetes.io/projected/e8b62c5c-8046-482a-a6d8-53f957838f39-kube-api-access-258b4\") pod \"test-curl-6900-38bc00\" (UID: \"e8b62c5c-8046-482a-a6d8-53f957838f39\") " pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:18.778398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:18.778297    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:19.088475 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:19.088430    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8b62c5c_8046_482a_a6d8_53f957838f39.slice/crio-913504a7df410eb6f8a81e9d8f2b271da72ec693d43fb78615c0f25b076348b0 WatchSource:0}: Error finding container 913504a7df410eb6f8a81e9d8f2b271da72ec693d43fb78615c0f25b076348b0: Status 404 returned error can't find the container with id 913504a7df410eb6f8a81e9d8f2b271da72ec693d43fb78615c0f25b076348b0
Nov 02 15:19:19.088894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:19.088870    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6900-38bc00"]
Nov 02 15:19:19.778806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:19.778765    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:20.065309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:20.065225    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:19:20.108157 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:20.108119    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6900-38bc00" event={"ID":"e8b62c5c-8046-482a-a6d8-53f957838f39","Type":"ContainerStarted","Data":"913504a7df410eb6f8a81e9d8f2b271da72ec693d43fb78615c0f25b076348b0"}
Nov 02 15:19:21.255899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:21.255866    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:21.422956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:21.422908    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:21.615717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:21.613581    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"]
Nov 02 15:19:21.623667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:21.618779    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy" containerID="cri-o://7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607" gracePeriod=30
Nov 02 15:19:21.632287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:21.628501    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:21.818264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:21.818225    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:22.456090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:22.456059    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:22.710498 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:22.710456    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:19:22.850302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:22.850265    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"]
Nov 02 15:19:22.855297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:22.855260    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:22.884896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:22.884861    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"]
Nov 02 15:19:22.903035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:22.902965    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-55e4c3-openshift-default-dockercfg-8jqw4\""
Nov 02 15:19:23.053726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.051068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:23.089669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.088169    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-podinfo\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.089669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.088287    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-credential-socket\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.089669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.088367    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-data\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.089669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.088436    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4j6\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-kube-api-access-vs4j6\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.089669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.088619    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-envoy\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.112673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.110435    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istiod-ca-cert\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.112673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.110526    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-socket\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.112673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.110736    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-token\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.112673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.110828    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-certs\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.203906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.202891    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6900-38bc00" event={"ID":"e8b62c5c-8046-482a-a6d8-53f957838f39","Type":"ContainerStarted","Data":"f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d"}
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.220839    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-data\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.220947    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4j6\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-kube-api-access-vs4j6\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.221302    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-envoy\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.221364    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istiod-ca-cert\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.221640    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-socket\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.221788    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-token\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.221883    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-certs\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.222187    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-podinfo\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.228799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.222264    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-credential-socket\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.237688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.235388    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-socket\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.237865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.237828    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-certs\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.246671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.239578    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istiod-ca-cert\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.246671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.242502    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-data\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.269849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.264994    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6900-38bc00" podStartSLOduration=2.000316077 podStartE2EDuration="5.264973265s" totalImagesPullingTime="3.264657188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:23.259964007 +0000 UTC m=+1357.513917634" watchObservedRunningTime="2026-11-02 15:19:23.264973265 +0000 UTC m=+1357.518926892"
Nov 02 15:19:23.269849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.268005    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-credential-socket\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.311871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.311748    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-envoy\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.314666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.312205    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-podinfo\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.314666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.313342    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4j6\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-kube-api-access-vs4j6\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.317125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.317089    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-token\") pod \"e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") " pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.379311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.379269    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:23.474214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.474186    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:23.628215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.628117    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:23.841724 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:23.841694    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c27a52_60bb_46a5_b431_0d231bfaf36a.slice/crio-2108832fc27bc4c7b3584900b901f2373651c0752ff3417fe9432fcb8580f5a4 WatchSource:0}: Error finding container 2108832fc27bc4c7b3584900b901f2373651c0752ff3417fe9432fcb8580f5a4: Status 404 returned error can't find the container with id 2108832fc27bc4c7b3584900b901f2373651c0752ff3417fe9432fcb8580f5a4
Nov 02 15:19:23.844497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.844468    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"]
Nov 02 15:19:23.844593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.844552    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:23.844636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.844616    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:23.844692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:23.844665    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:24.286126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.284240    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" event={"ID":"e7c27a52-60bb-46a5-b431-0d231bfaf36a","Type":"ContainerStarted","Data":"ba743bb9b6c91e91893ff6ac1bf55fa01653c4c533cc64af65bb7f1a46219112"}
Nov 02 15:19:24.286126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.284330    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" event={"ID":"e7c27a52-60bb-46a5-b431-0d231bfaf36a","Type":"ContainerStarted","Data":"2108832fc27bc4c7b3584900b901f2373651c0752ff3417fe9432fcb8580f5a4"}
Nov 02 15:19:24.352678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.352309    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podStartSLOduration=2.352288205 podStartE2EDuration="2.352288205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:24.350447837 +0000 UTC m=+1358.604401462" watchObservedRunningTime="2026-11-02 15:19:24.352288205 +0000 UTC m=+1358.606241830"
Nov 02 15:19:24.499806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.496397    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:24.506698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.503858    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 15:19:24.506698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.503934    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 15:19:24.620771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.620595    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6900-38bc00"]
Nov 02 15:19:24.678671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:24.674200    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6900-38bc00"]
Nov 02 15:19:25.167429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.167391    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"]
Nov 02 15:19:25.168173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.168150    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" containerName="test-curl-6900-38bc00"
Nov 02 15:19:25.168173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.168170    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" containerName="test-curl-6900-38bc00"
Nov 02 15:19:25.168387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.168306    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" containerName="test-curl-6900-38bc00"
Nov 02 15:19:25.172375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.172357    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.190337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.190311    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:19:25.190802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.190775    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:19:25.193911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.193886    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"]
Nov 02 15:19:25.241007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.240958    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjt2\" (UniqueName: \"kubernetes.io/projected/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-kube-api-access-7cjt2\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.241213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.241020    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.241408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.241385    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.290915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.290863    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6900-38bc00" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" containerName="test-curl-6900-38bc00" containerID="cri-o://f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d" gracePeriod=2
Nov 02 15:19:25.306554 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:25.306517    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-38bc00\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:25.342701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.342641    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.342893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.342768    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjt2\" (UniqueName: \"kubernetes.io/projected/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-kube-api-access-7cjt2\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.342893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.342807    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.344078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.344037    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.345430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.345406    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.368880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.368846    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjt2\" (UniqueName: \"kubernetes.io/projected/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-kube-api-access-7cjt2\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.494522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.493226    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:25.517282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.511208    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:25.587728 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:25.584850    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-38bc00\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:25.874814 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:25.874777    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-38bc00\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:25.936707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.936612    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"]
Nov 02 15:19:25.940166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:25.940123    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:26.309386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:26.309349    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x" event={"ID":"7420f4a2-e22d-4719-ba7b-59958a3b5ec2","Type":"ContainerStarted","Data":"57daad4007d8a500ce7e712a0f82c273a8b00740eafd60a9183a6b4cee410517"}
Nov 02 15:19:26.311119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:26.311094    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:26.315919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:26.315894    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:19:27.116671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.103445    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149417    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istiod-ca-cert\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149457    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-token\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-token\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149485    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-podinfo\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149506    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-envoy\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149526    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-certs\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149656    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-socket\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149763    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-kube-api-access-p2kq9\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-kube-api-access-p2kq9\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149771    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istiod-ca-cert" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149856    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-credential-socket\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.149887    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-data\") pod \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\" (UID: \"d30f3e5b-5f21-4f8e-ab39-3072f804a77d\") "
Nov 02 15:19:27.150984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.150355    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.150984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.150539    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-data" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:27.150984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.150704    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-certs" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:27.150984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.150818    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-socket" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:27.151259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.151157    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-credential-socket" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:27.151838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.151810    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-envoy" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:19:27.152547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.152512    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-token" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:27.152946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.152924    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-podinfo" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:19:27.153406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.153381    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-kube-api-access-p2kq9" pod "d30f3e5b-5f21-4f8e-ab39-3072f804a77d" (UID: "d30f3e5b-5f21-4f8e-ab39-3072f804a77d"). InnerVolumeSpecName "kube-api-access-p2kq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:27.253126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.253091    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.253328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.253313    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2kq9\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-kube-api-access-p2kq9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.253434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.253421    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.253519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.253508    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.253620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.253609    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.253890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.253866    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.254115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.254065    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.254225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.254212    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30f3e5b-5f21-4f8e-ab39-3072f804a77d-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:27.358059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.357910    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x" event={"ID":"7420f4a2-e22d-4719-ba7b-59958a3b5ec2","Type":"ContainerStarted","Data":"291c234197905045a208f6b0a2b2ac9d7f27e7376df32a76bfa14c789b99458a"}
Nov 02 15:19:27.362264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.362001    7019 generic.go:356] "Generic (PLEG): container finished" podID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerID="7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607" exitCode=0
Nov 02 15:19:27.362392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.362139    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"
Nov 02 15:19:27.362821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.362158    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" event={"ID":"d30f3e5b-5f21-4f8e-ab39-3072f804a77d","Type":"ContainerDied","Data":"7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607"}
Nov 02 15:19:27.363582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.363562    7019 scope.go:122] "RemoveContainer" containerID="7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607"
Nov 02 15:19:27.364518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.364458    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk" event={"ID":"d30f3e5b-5f21-4f8e-ab39-3072f804a77d","Type":"ContainerDied","Data":"bce0a10732ad89ff9c28b521d8b3e342c728ccd2563d7bfb6c2cb782d50f0ceb"}
Nov 02 15:19:27.453710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.444063    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x" podStartSLOduration=2.444041436 podStartE2EDuration="2.444041436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:27.439077334 +0000 UTC m=+1361.693030969" watchObservedRunningTime="2026-11-02 15:19:27.444041436 +0000 UTC m=+1361.697995061"
Nov 02 15:19:27.458181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.457003    7019 scope.go:122] "RemoveContainer" containerID="864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412"
Nov 02 15:19:27.547443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.539698    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"]
Nov 02 15:19:27.551444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.551406    7019 scope.go:122] "RemoveContainer" containerID="7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607"
Nov 02 15:19:27.552003 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:27.551978    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607\": container with ID starting with 7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607 not found: ID does not exist" containerID="7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607"
Nov 02 15:19:27.552237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.552211    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607"} err="failed to get container status \"7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607\": rpc error: code = NotFound desc = could not find container \"7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607\": container with ID starting with 7670043b86aadb3aad6dfa48c826d8692ecccdd096a852e85de867f5c5e57607 not found: ID does not exist"
Nov 02 15:19:27.552343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.552332    7019 scope.go:122] "RemoveContainer" containerID="864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412"
Nov 02 15:19:27.552846 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:27.552788    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412\": container with ID starting with 864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412 not found: ID does not exist" containerID="864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412"
Nov 02 15:19:27.552846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.552820    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412"} err="failed to get container status \"864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412\": rpc error: code = NotFound desc = could not find container \"864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412\": container with ID starting with 864ebb4deada467f8b3cdce0cf99d4911fb06b69f0123281cd7f4d037cbd2412 not found: ID does not exist"
Nov 02 15:19:27.553996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.553823    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c7afd1a6-openshift-default-78859757d9-bb6nk"]
Nov 02 15:19:27.962538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:27.962477    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" path="/var/lib/kubelet/pods/d30f3e5b-5f21-4f8e-ab39-3072f804a77d/volumes"
Nov 02 15:19:28.030334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.030136    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:28.062009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.061956    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b62c5c-8046-482a-a6d8-53f957838f39-kube-api-access-258b4\" (UniqueName: \"kubernetes.io/projected/e8b62c5c-8046-482a-a6d8-53f957838f39-kube-api-access-258b4\") pod \"e8b62c5c-8046-482a-a6d8-53f957838f39\" (UID: \"e8b62c5c-8046-482a-a6d8-53f957838f39\") "
Nov 02 15:19:28.064129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.064095    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b62c5c-8046-482a-a6d8-53f957838f39-kube-api-access-258b4" pod "e8b62c5c-8046-482a-a6d8-53f957838f39" (UID: "e8b62c5c-8046-482a-a6d8-53f957838f39"). InnerVolumeSpecName "kube-api-access-258b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:28.163943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.163905    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258b4\" (UniqueName: \"kubernetes.io/projected/e8b62c5c-8046-482a-a6d8-53f957838f39-kube-api-access-258b4\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:28.374360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.374323    7019 generic.go:356] "Generic (PLEG): container finished" podID="e8b62c5c-8046-482a-a6d8-53f957838f39" containerID="f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d" exitCode=137
Nov 02 15:19:28.374579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.374415    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:28.374579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.374450    7019 scope.go:122] "RemoveContainer" containerID="f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d"
Nov 02 15:19:28.377383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.377352    7019 generic.go:356] "Generic (PLEG): container finished" podID="7420f4a2-e22d-4719-ba7b-59958a3b5ec2" containerID="291c234197905045a208f6b0a2b2ac9d7f27e7376df32a76bfa14c789b99458a" exitCode=0
Nov 02 15:19:28.377713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.377632    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x" event={"ID":"7420f4a2-e22d-4719-ba7b-59958a3b5ec2","Type":"ContainerDied","Data":"291c234197905045a208f6b0a2b2ac9d7f27e7376df32a76bfa14c789b99458a"}
Nov 02 15:19:28.393243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.393224    7019 scope.go:122] "RemoveContainer" containerID="f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d"
Nov 02 15:19:28.393622 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:28.393589    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d\": container with ID starting with f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d not found: ID does not exist" containerID="f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d"
Nov 02 15:19:28.393775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:28.393633    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d"} err="failed to get container status \"f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d\": rpc error: code = NotFound desc = could not find container \"f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d\": container with ID starting with f5ec2c0da8f17cc176fd3ba02e825620da6fb9e306f74dc568eb49a9f821622d not found: ID does not exist"
Nov 02 15:19:28.439692 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:28.437232    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-38bc00\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:29.156119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.156070    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:29.316814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.316779    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:29.481688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.478779    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:29.539158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.539006    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"]
Nov 02 15:19:29.539780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.539760    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy"
Nov 02 15:19:29.539883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.539781    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy"
Nov 02 15:19:29.539883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.539798    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy"
Nov 02 15:19:29.539883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.539804    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy"
Nov 02 15:19:29.540065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.539936    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy"
Nov 02 15:19:29.540065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.539957    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30f3e5b-5f21-4f8e-ab39-3072f804a77d" containerName="istio-proxy"
Nov 02 15:19:29.545124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.545102    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:29.560600 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:29.560565    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-38bc00\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:29.561817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.561794    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-55e4c3\""
Nov 02 15:19:29.564514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.564488    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"]
Nov 02 15:19:29.569261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.569235    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:29.571371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.571338    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-55e4c3-dockercfg-jkjbg\""
Nov 02 15:19:29.591395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.591354    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-kube-api-access-7cjt2\" (UniqueName: \"kubernetes.io/projected/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-kube-api-access-7cjt2\") pod \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") "
Nov 02 15:19:29.591563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.591432    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-openshift-ca\") pod \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") "
Nov 02 15:19:29.591563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.591550    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-xks-ca\") pod \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\" (UID: \"7420f4a2-e22d-4719-ba7b-59958a3b5ec2\") "
Nov 02 15:19:29.592097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.592031    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31488e9e-b222-49bb-898b-80aeac7e0481-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:29.592221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.592188    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bccms\" (UniqueName: \"kubernetes.io/projected/31488e9e-b222-49bb-898b-80aeac7e0481-kube-api-access-bccms\") pod \"payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:29.594166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.594136    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-openshift-ca" pod "7420f4a2-e22d-4719-ba7b-59958a3b5ec2" (UID: "7420f4a2-e22d-4719-ba7b-59958a3b5ec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:29.595739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.595707    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-kube-api-access-7cjt2" pod "7420f4a2-e22d-4719-ba7b-59958a3b5ec2" (UID: "7420f4a2-e22d-4719-ba7b-59958a3b5ec2"). InnerVolumeSpecName "kube-api-access-7cjt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:29.596990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.596949    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-xks-ca" pod "7420f4a2-e22d-4719-ba7b-59958a3b5ec2" (UID: "7420f4a2-e22d-4719-ba7b-59958a3b5ec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:29.599229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.599204    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"]
Nov 02 15:19:29.600010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.599992    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7420f4a2-e22d-4719-ba7b-59958a3b5ec2" containerName="revoke-keys"
Nov 02 15:19:29.600105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.600012    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7420f4a2-e22d-4719-ba7b-59958a3b5ec2" containerName="revoke-keys"
Nov 02 15:19:29.600250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.600236    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7420f4a2-e22d-4719-ba7b-59958a3b5ec2" containerName="revoke-keys"
Nov 02 15:19:29.604485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.604460    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:29.624175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.624141    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"]
Nov 02 15:19:29.873682 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:29.871870    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-38bc00\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" pod="opendatahub/test-curl-6900-38bc00"
Nov 02 15:19:29.935068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.935026    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b62c5c-8046-482a-a6d8-53f957838f39" path="/var/lib/kubelet/pods/e8b62c5c-8046-482a-a6d8-53f957838f39/volumes"
Nov 02 15:19:29.953694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.721633    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da826e5-1db1-4875-b9ce-5354e2e896be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") " pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:29.954201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.953895    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvr4x\" (UniqueName: \"kubernetes.io/projected/2da826e5-1db1-4875-b9ce-5354e2e896be-kube-api-access-hvr4x\") pod \"payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") " pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:29.954201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.954131    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31488e9e-b222-49bb-898b-80aeac7e0481-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:29.954375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.954224    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bccms\" (UniqueName: \"kubernetes.io/projected/31488e9e-b222-49bb-898b-80aeac7e0481-kube-api-access-bccms\") pod \"payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:29.954440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.954426    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjt2\" (UniqueName: \"kubernetes.io/projected/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-kube-api-access-7cjt2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:29.954497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.954444    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:29.954497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.954461    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7420f4a2-e22d-4719-ba7b-59958a3b5ec2-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:29.955020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.954996    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31488e9e-b222-49bb-898b-80aeac7e0481-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:29.993477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:29.993436    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bccms\" (UniqueName: \"kubernetes.io/projected/31488e9e-b222-49bb-898b-80aeac7e0481-kube-api-access-bccms\") pod \"payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:30.056130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.056092    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da826e5-1db1-4875-b9ce-5354e2e896be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") " pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:30.056309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.056225    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvr4x\" (UniqueName: \"kubernetes.io/projected/2da826e5-1db1-4875-b9ce-5354e2e896be-kube-api-access-hvr4x\") pod \"payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") " pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:30.056923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.056900    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da826e5-1db1-4875-b9ce-5354e2e896be-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") " pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:30.081180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.081152    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvr4x\" (UniqueName: \"kubernetes.io/projected/2da826e5-1db1-4875-b9ce-5354e2e896be-kube-api-access-hvr4x\") pod \"payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") " pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:30.181660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.181568    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:30.356612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.356574    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:30.402363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.402326    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x" event={"ID":"7420f4a2-e22d-4719-ba7b-59958a3b5ec2","Type":"ContainerDied","Data":"57daad4007d8a500ce7e712a0f82c273a8b00740eafd60a9183a6b4cee410517"}
Nov 02 15:19:30.402363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.402360    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"
Nov 02 15:19:30.402589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.402422    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57daad4007d8a500ce7e712a0f82c273a8b00740eafd60a9183a6b4cee410517"
Nov 02 15:19:30.432482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.432325    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:30.532991 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:30.532949    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31488e9e_b222_49bb_898b_80aeac7e0481.slice/crio-3cf38f17e261db56af9eafe749d248eb7fbdb94b6bf78b6254b38e4f106b1827 WatchSource:0}: Error finding container 3cf38f17e261db56af9eafe749d248eb7fbdb94b6bf78b6254b38e4f106b1827: Status 404 returned error can't find the container with id 3cf38f17e261db56af9eafe749d248eb7fbdb94b6bf78b6254b38e4f106b1827
Nov 02 15:19:30.536532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.536502    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"]
Nov 02 15:19:30.709786 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:30.709749    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da826e5_1db1_4875_b9ce_5354e2e896be.slice/crio-5cb79d71c515d4c68b3f435e69e2666366941fb9c3c24ae3369562999096d9f0 WatchSource:0}: Error finding container 5cb79d71c515d4c68b3f435e69e2666366941fb9c3c24ae3369562999096d9f0: Status 404 returned error can't find the container with id 5cb79d71c515d4c68b3f435e69e2666366941fb9c3c24ae3369562999096d9f0
Nov 02 15:19:30.714012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:30.713988    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"]
Nov 02 15:19:31.411732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.411687    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br" event={"ID":"31488e9e-b222-49bb-898b-80aeac7e0481","Type":"ContainerStarted","Data":"e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0"}
Nov 02 15:19:31.412207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.411757    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br" event={"ID":"31488e9e-b222-49bb-898b-80aeac7e0481","Type":"ContainerStarted","Data":"3cf38f17e261db56af9eafe749d248eb7fbdb94b6bf78b6254b38e4f106b1827"}
Nov 02 15:19:31.412207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.411898    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:31.413503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.413477    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2" event={"ID":"2da826e5-1db1-4875-b9ce-5354e2e896be","Type":"ContainerStarted","Data":"6641dd91ccb8ec996b98039642a325795d458e4cfbc3412b390640a466de1195"}
Nov 02 15:19:31.413640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.413521    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2" event={"ID":"2da826e5-1db1-4875-b9ce-5354e2e896be","Type":"ContainerStarted","Data":"5cb79d71c515d4c68b3f435e69e2666366941fb9c3c24ae3369562999096d9f0"}
Nov 02 15:19:31.414168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.414150    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:31.460246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.460190    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br" podStartSLOduration=2.46017446 podStartE2EDuration="2.46017446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:31.45912122 +0000 UTC m=+1365.713074845" watchObservedRunningTime="2026-11-02 15:19:31.46017446 +0000 UTC m=+1365.714128127"
Nov 02 15:19:31.504295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:31.504243    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2" podStartSLOduration=2.504228532 podStartE2EDuration="2.504228532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:31.503704501 +0000 UTC m=+1365.757658123" watchObservedRunningTime="2026-11-02 15:19:31.504228532 +0000 UTC m=+1365.758182156"
Nov 02 15:19:32.279526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:32.279495    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:34.095669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:34.095612    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:35.790073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:35.790033    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:35.953062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:35.953022    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:36.363266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.363233    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:36.719230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.719139    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"]
Nov 02 15:19:36.727535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.727509    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.754965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.754931    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"]
Nov 02 15:19:36.761122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.761095    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-78d075-openshift-default-dockercfg-5d2sq\""
Nov 02 15:19:36.808337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.808300    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qt67\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-kube-api-access-7qt67\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.809026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.809000    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf5e56a-3578-433f-9169-a1901aeb6de6-istiod-ca-cert\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.809225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.809210    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-credential-socket\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.809386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.809370    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-token\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.809626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.809611    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-certs\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.809843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.809828    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-podinfo\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.810103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.810082    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-socket\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.810289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.810273    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-envoy\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.810430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.810414    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-data\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.814673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.814633    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.912714    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-socket\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.912798    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-envoy\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.912834    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-data\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.912932    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qt67\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-kube-api-access-7qt67\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.913004    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf5e56a-3578-433f-9169-a1901aeb6de6-istiod-ca-cert\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.913032    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-credential-socket\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.913060    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-token\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.913164    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-certs\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.913670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.913212    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-podinfo\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.929673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.914447    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-socket\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.929673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.926363    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-envoy\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.929673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.926634    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-data\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.929673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.926886    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-credential-socket\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.929673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.927519    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf5e56a-3578-433f-9169-a1901aeb6de6-istiod-ca-cert\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.929673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.928145    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-certs\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.955687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.938824    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-podinfo\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.973761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.959329    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qt67\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-kube-api-access-7qt67\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:36.977719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.975500    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:36.977719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:36.976194    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-token\") pod \"e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") " pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:37.058687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:37.050297    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:37.520222 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:37.520183    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf5e56a_3578_433f_9169_a1901aeb6de6.slice/crio-ff2cf1dd471cb9c035138c84bea299308aa13502269c4e3d2fd3388a57005ab0 WatchSource:0}: Error finding container ff2cf1dd471cb9c035138c84bea299308aa13502269c4e3d2fd3388a57005ab0: Status 404 returned error can't find the container with id ff2cf1dd471cb9c035138c84bea299308aa13502269c4e3d2fd3388a57005ab0
Nov 02 15:19:37.520640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:37.520619    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"]
Nov 02 15:19:37.523039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:37.523001    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:37.523118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:37.523071    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:37.523118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:37.523100    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:37.676694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:37.676581    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:38.513228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:38.513189    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" event={"ID":"4bf5e56a-3578-433f-9169-a1901aeb6de6","Type":"ContainerStarted","Data":"d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87"}
Nov 02 15:19:38.513750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:38.513267    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" event={"ID":"4bf5e56a-3578-433f-9169-a1901aeb6de6","Type":"ContainerStarted","Data":"ff2cf1dd471cb9c035138c84bea299308aa13502269c4e3d2fd3388a57005ab0"}
Nov 02 15:19:38.579297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:38.578546    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podStartSLOduration=2.578513083 podStartE2EDuration="2.578513083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:38.576954932 +0000 UTC m=+1372.830908557" watchObservedRunningTime="2026-11-02 15:19:38.578513083 +0000 UTC m=+1372.832466708"
Nov 02 15:19:39.051342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:39.051302    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:39.057148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:39.057123    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:39.490028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:39.489991    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:39.529095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:39.529029    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:39.530508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:39.530482    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:19:41.004231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:41.004184    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:41.160663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:41.160597    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:41.376730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:41.376122    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:42.427103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.427069    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:19:42.855770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.855677    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:42.948426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.948389    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"]
Nov 02 15:19:42.953203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.953178    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:42.962093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.961770    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4209db8-105a-451a-9a43-719fe1a2a29b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:42.962093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.961902    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7kff\" (UniqueName: \"kubernetes.io/projected/d4209db8-105a-451a-9a43-719fe1a2a29b-kube-api-access-s7kff\") pod \"payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:42.971971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.971944    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-78d075\""
Nov 02 15:19:42.975957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:42.975932    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"]
Nov 02 15:19:43.057096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.036676    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-78d075-dockercfg-4mtqn\""
Nov 02 15:19:43.135044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.066508    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:43.135044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.092875    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4209db8-105a-451a-9a43-719fe1a2a29b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:43.135044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.132799    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7kff\" (UniqueName: \"kubernetes.io/projected/d4209db8-105a-451a-9a43-719fe1a2a29b-kube-api-access-s7kff\") pod \"payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:43.135044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.093738    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4209db8-105a-451a-9a43-719fe1a2a29b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:43.285456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.284258    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"]
Nov 02 15:19:43.300802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.300744    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7kff\" (UniqueName: \"kubernetes.io/projected/d4209db8-105a-451a-9a43-719fe1a2a29b-kube-api-access-s7kff\") pod \"payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:43.301723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.301688    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:43.330084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.329181    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:43.397537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.397451    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"]
Nov 02 15:19:43.397908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.397886    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.442933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.438738    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:19:43.467719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.467640    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c214d04-6f64-4bea-b8e4-e98b175a8e44-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.477709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.467952    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/8c214d04-6f64-4bea-b8e4-e98b175a8e44-kube-api-access-2l8sf\") pod \"payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.573694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.573608    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/8c214d04-6f64-4bea-b8e4-e98b175a8e44-kube-api-access-2l8sf\") pod \"payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.580824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.578968    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c214d04-6f64-4bea-b8e4-e98b175a8e44-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.580824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.580381    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c214d04-6f64-4bea-b8e4-e98b175a8e44-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.603040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.603004    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/8c214d04-6f64-4bea-b8e4-e98b175a8e44-kube-api-access-2l8sf\") pod \"payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.724078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.723987    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:43.752046 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:43.752018    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4209db8_105a_451a_9a43_719fe1a2a29b.slice/crio-dc2380d2d0b3e9ff148f2b61b026dc86f7eb027f04e11e56e65960c26ab53bfc WatchSource:0}: Error finding container dc2380d2d0b3e9ff148f2b61b026dc86f7eb027f04e11e56e65960c26ab53bfc: Status 404 returned error can't find the container with id dc2380d2d0b3e9ff148f2b61b026dc86f7eb027f04e11e56e65960c26ab53bfc
Nov 02 15:19:43.752479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:43.752457    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"]
Nov 02 15:19:44.040665 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:44.040614    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c214d04_6f64_4bea_b8e4_e98b175a8e44.slice/crio-aadb4d4a34028fdf08c483e2d18d1cfdf97f6bce226d87541761ea8b58420d03 WatchSource:0}: Error finding container aadb4d4a34028fdf08c483e2d18d1cfdf97f6bce226d87541761ea8b58420d03: Status 404 returned error can't find the container with id aadb4d4a34028fdf08c483e2d18d1cfdf97f6bce226d87541761ea8b58420d03
Nov 02 15:19:44.040826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.040806    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"]
Nov 02 15:19:44.580272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.580231    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" event={"ID":"8c214d04-6f64-4bea-b8e4-e98b175a8e44","Type":"ContainerStarted","Data":"7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d"}
Nov 02 15:19:44.580768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.580300    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" event={"ID":"8c214d04-6f64-4bea-b8e4-e98b175a8e44","Type":"ContainerStarted","Data":"aadb4d4a34028fdf08c483e2d18d1cfdf97f6bce226d87541761ea8b58420d03"}
Nov 02 15:19:44.580768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.580519    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:44.583061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.583029    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" event={"ID":"d4209db8-105a-451a-9a43-719fe1a2a29b","Type":"ContainerStarted","Data":"17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a"}
Nov 02 15:19:44.583209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.583099    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" event={"ID":"d4209db8-105a-451a-9a43-719fe1a2a29b","Type":"ContainerStarted","Data":"dc2380d2d0b3e9ff148f2b61b026dc86f7eb027f04e11e56e65960c26ab53bfc"}
Nov 02 15:19:44.583209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.583125    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:44.624821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.624764    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" podStartSLOduration=2.624749257 podStartE2EDuration="2.624749257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:44.623905919 +0000 UTC m=+1378.877859556" watchObservedRunningTime="2026-11-02 15:19:44.624749257 +0000 UTC m=+1378.878702881"
Nov 02 15:19:44.672352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.672292    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" podStartSLOduration=2.67227797 podStartE2EDuration="2.67227797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:44.667822674 +0000 UTC m=+1378.921776300" watchObservedRunningTime="2026-11-02 15:19:44.67227797 +0000 UTC m=+1378.926231594"
Nov 02 15:19:44.788461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:44.788425    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:46.032711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:46.032677    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:46.334885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:46.334773    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:47.314370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:47.314328    7019 scope.go:122] "RemoveContainer" containerID="f592f4868dfd2df48f355ea0bf8a1dea1f26de01c88c44da26844eba043059c2"
Nov 02 15:19:47.326449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:47.326426    7019 scope.go:122] "RemoveContainer" containerID="baaf9ed0060bfaf46aead267517b20e54ee4aa2bf47b15c790457b2e23fc47a9"
Nov 02 15:19:48.038285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:48.038246    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:49.007677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:49.006693    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:19:50.088823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:50.088788    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:50.773382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:50.773148    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:51.241041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.241004    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:51.664724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.664623    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"]
Nov 02 15:19:51.696554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.695554    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"]
Nov 02 15:19:51.696554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.695931    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.721923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.721897    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"]
Nov 02 15:19:51.722264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.722237    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"]
Nov 02 15:19:51.723724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.722452    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:51.741699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.740716    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.753450    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-certs\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.753573    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.753688    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-credential-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.753718    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd66q\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-kube-api-access-gd66q\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.753760    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-data\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.753831    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-envoy\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.753968    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-token\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.754003    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dde604-430f-4f5f-bae4-111c194a1d2b-istiod-ca-cert\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.761424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.754027    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-podinfo\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.792723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.790034    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2bbcb5-openshift-default-dockercfg-m5gt8\""
Nov 02 15:19:51.792723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.790120    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:19:51.792723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.790223    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.872520    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-credential-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.872577    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd66q\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-kube-api-access-gd66q\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.872622    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-data\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.872718    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-envoy\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.872864    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-token\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.872898    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dde604-430f-4f5f-bae4-111c194a1d2b-istiod-ca-cert\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.872925    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-podinfo\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.873014    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-certs\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.873055    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874824ca-f99e-453d-8eee-58c708804f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.873121    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874824ca-f99e-453d-8eee-58c708804f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.873189    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.873244    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hgc\" (UniqueName: \"kubernetes.io/projected/874824ca-f99e-453d-8eee-58c708804f96-kube-api-access-h8hgc\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:51.875672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.873503    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-credential-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.895037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.894458    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-certs\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.895037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.894734    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-socket\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.895531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.895505    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-data\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.896285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.896243    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dde604-430f-4f5f-bae4-111c194a1d2b-istiod-ca-cert\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:51.963311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.947109    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-token\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:52.009264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.989386    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874824ca-f99e-453d-8eee-58c708804f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:52.049672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.048980    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874824ca-f99e-453d-8eee-58c708804f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:52.049672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.049136    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hgc\" (UniqueName: \"kubernetes.io/projected/874824ca-f99e-453d-8eee-58c708804f96-kube-api-access-h8hgc\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:52.049672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.991066    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874824ca-f99e-453d-8eee-58c708804f96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:52.049971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:51.990172    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-podinfo\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:52.050303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.045772    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-envoy\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:52.051627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.051605    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd66q\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-kube-api-access-gd66q\") pod \"e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") " pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:52.054754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.054687    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874824ca-f99e-453d-8eee-58c708804f96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:52.085349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.085280    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hgc\" (UniqueName: \"kubernetes.io/projected/874824ca-f99e-453d-8eee-58c708804f96-kube-api-access-h8hgc\") pod \"maas-api-revoke-sub-bd06887a-tb46l\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:52.111327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.111293    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:52.280328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.280292    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:52.324588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.324541    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:52.521795 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:52.521762    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874824ca_f99e_453d_8eee_58c708804f96.slice/crio-9b27761f97f515a5b5d77873b52a8d0052da62270f0111250031af0ba796a557 WatchSource:0}: Error finding container 9b27761f97f515a5b5d77873b52a8d0052da62270f0111250031af0ba796a557: Status 404 returned error can't find the container with id 9b27761f97f515a5b5d77873b52a8d0052da62270f0111250031af0ba796a557
Nov 02 15:19:52.525632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.525299    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"]
Nov 02 15:19:52.554616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.551958    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:52.662879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.662841    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l" event={"ID":"874824ca-f99e-453d-8eee-58c708804f96","Type":"ContainerStarted","Data":"9b27761f97f515a5b5d77873b52a8d0052da62270f0111250031af0ba796a557"}
Nov 02 15:19:52.716575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.716545    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"]
Nov 02 15:19:52.719228 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:19:52.719202    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62dde604_430f_4f5f_bae4_111c194a1d2b.slice/crio-abc557bbbc159a398f6e1380816a0b999a40618535b6213290c5c78ddbb3bfdf WatchSource:0}: Error finding container abc557bbbc159a398f6e1380816a0b999a40618535b6213290c5c78ddbb3bfdf: Status 404 returned error can't find the container with id abc557bbbc159a398f6e1380816a0b999a40618535b6213290c5c78ddbb3bfdf
Nov 02 15:19:52.722984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.722948    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:52.723071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.723018    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:52.723071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:52.723044    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:19:52.767190 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:19:52.767114    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:19:53.675852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:53.675812    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" event={"ID":"62dde604-430f-4f5f-bae4-111c194a1d2b","Type":"ContainerStarted","Data":"fddb1e4877332ad6835a7f515ef91326f53adf852fc83bb763a55975c5bc6077"}
Nov 02 15:19:53.676370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:53.675881    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" event={"ID":"62dde604-430f-4f5f-bae4-111c194a1d2b","Type":"ContainerStarted","Data":"abc557bbbc159a398f6e1380816a0b999a40618535b6213290c5c78ddbb3bfdf"}
Nov 02 15:19:53.678082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:53.678050    7019 generic.go:356] "Generic (PLEG): container finished" podID="874824ca-f99e-453d-8eee-58c708804f96" containerID="e988464b0a22bfad15015a20508d856615f8a2dc5a7e44e87e6fa3585912efe3" exitCode=0
Nov 02 15:19:53.678195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:53.678104    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l" event={"ID":"874824ca-f99e-453d-8eee-58c708804f96","Type":"ContainerDied","Data":"e988464b0a22bfad15015a20508d856615f8a2dc5a7e44e87e6fa3585912efe3"}
Nov 02 15:19:53.740605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:53.740483    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podStartSLOduration=2.7404646870000002 podStartE2EDuration="2.740464687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:53.737960267 +0000 UTC m=+1387.991913892" watchObservedRunningTime="2026-11-02 15:19:53.740464687 +0000 UTC m=+1387.994418312"
Nov 02 15:19:54.050003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.047766    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:54.325217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.325116    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:54.331129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.331101    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:54.686828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.686792    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:54.688880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.688847    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:19:54.868364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.868338    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:54.929598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.929563    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874824ca-f99e-453d-8eee-58c708804f96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874824ca-f99e-453d-8eee-58c708804f96-maas-api-openshift-ca\") pod \"874824ca-f99e-453d-8eee-58c708804f96\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") "
Nov 02 15:19:54.929805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.929664    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874824ca-f99e-453d-8eee-58c708804f96-kube-api-access-h8hgc\" (UniqueName: \"kubernetes.io/projected/874824ca-f99e-453d-8eee-58c708804f96-kube-api-access-h8hgc\") pod \"874824ca-f99e-453d-8eee-58c708804f96\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") "
Nov 02 15:19:54.929939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.929926    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874824ca-f99e-453d-8eee-58c708804f96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874824ca-f99e-453d-8eee-58c708804f96-maas-api-xks-ca\") pod \"874824ca-f99e-453d-8eee-58c708804f96\" (UID: \"874824ca-f99e-453d-8eee-58c708804f96\") "
Nov 02 15:19:54.929989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.929951    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874824ca-f99e-453d-8eee-58c708804f96-maas-api-openshift-ca" pod "874824ca-f99e-453d-8eee-58c708804f96" (UID: "874824ca-f99e-453d-8eee-58c708804f96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:54.930567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.930546    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874824ca-f99e-453d-8eee-58c708804f96-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:54.931830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.931807    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874824ca-f99e-453d-8eee-58c708804f96-kube-api-access-h8hgc" pod "874824ca-f99e-453d-8eee-58c708804f96" (UID: "874824ca-f99e-453d-8eee-58c708804f96"). InnerVolumeSpecName "kube-api-access-h8hgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:54.931945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:54.931892    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874824ca-f99e-453d-8eee-58c708804f96-maas-api-xks-ca" pod "874824ca-f99e-453d-8eee-58c708804f96" (UID: "874824ca-f99e-453d-8eee-58c708804f96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:55.032575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.032538    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874824ca-f99e-453d-8eee-58c708804f96-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:55.032575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.032568    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hgc\" (UniqueName: \"kubernetes.io/projected/874824ca-f99e-453d-8eee-58c708804f96-kube-api-access-h8hgc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:19:55.603311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.603272    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:19:55.603470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.603343    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:19:55.699405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.699370    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l" event={"ID":"874824ca-f99e-453d-8eee-58c708804f96","Type":"ContainerDied","Data":"9b27761f97f515a5b5d77873b52a8d0052da62270f0111250031af0ba796a557"}
Nov 02 15:19:55.699889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.699435    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b27761f97f515a5b5d77873b52a8d0052da62270f0111250031af0ba796a557"
Nov 02 15:19:55.699889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.699381    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"
Nov 02 15:19:55.822816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:55.822777    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:56.211599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:56.211566    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:56.376405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:56.376372    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:57.680412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:57.680374    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:57.854736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:57.853212    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:58.002621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.002422    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:58.112419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.112382    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"]
Nov 02 15:19:58.113596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.113573    7019 cpu_manager.go:414] "containerMap: removing container" podUID="874824ca-f99e-453d-8eee-58c708804f96" containerName="revoke-keys"
Nov 02 15:19:58.113730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.113599    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874824ca-f99e-453d-8eee-58c708804f96" containerName="revoke-keys"
Nov 02 15:19:58.113828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.113814    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="874824ca-f99e-453d-8eee-58c708804f96" containerName="revoke-keys"
Nov 02 15:19:58.122883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.122857    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.195671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.190533    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2bbcb5-dockercfg-vvtzx\""
Nov 02 15:19:58.195671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.190668    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2bbcb5\""
Nov 02 15:19:58.303724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.283859    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"]
Nov 02 15:19:58.303724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.285582    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftwh\" (UniqueName: \"kubernetes.io/projected/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-kube-api-access-tftwh\") pod \"payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.303724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.285689    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.414720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.408001    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftwh\" (UniqueName: \"kubernetes.io/projected/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-kube-api-access-tftwh\") pod \"payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.414720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.408136    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.414720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.409084    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.508672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.508350    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftwh\" (UniqueName: \"kubernetes.io/projected/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-kube-api-access-tftwh\") pod \"payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.540875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.540839    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:58.961089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:58.957236    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"]
Nov 02 15:19:59.669227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:59.669191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:19:59.738669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:59.735033    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" event={"ID":"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f","Type":"ContainerStarted","Data":"8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1"}
Nov 02 15:19:59.738669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:59.735128    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" event={"ID":"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f","Type":"ContainerStarted","Data":"a40aa1c015dfc4610561cd04bc79a3e7cf3d77ab51283b4fc807bb887e9e41da"}
Nov 02 15:19:59.738669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:59.735264    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:19:59.782597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:19:59.782480    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" podStartSLOduration=1.782459313 podStartE2EDuration="1.782459313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:59.779484078 +0000 UTC m=+1394.033437703" watchObservedRunningTime="2026-11-02 15:19:59.782459313 +0000 UTC m=+1394.036412940"
Nov 02 15:20:00.310365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:00.310300    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:01.104121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:01.104075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:01.765935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:01.765895    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:01.977219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:01.976877    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:02.010245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:02.010206    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"]
Nov 02 15:20:02.023566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:02.023482    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ad276f-6lqh6"]
Nov 02 15:20:02.138712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:02.138675    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:02.586976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:02.586930    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:03.875173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:03.875132    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929f4b30-e688-40f1-a9a1-3c993623cd4f" path="/var/lib/kubelet/pods/929f4b30-e688-40f1-a9a1-3c993623cd4f/volumes"
Nov 02 15:20:03.938938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:03.938903    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:05.573007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:05.572973    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:07.230702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:07.230341    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:08.615253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:08.615203    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:08.798129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:08.798087    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:10.006784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:10.006750    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:10.191406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:10.191362    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:10.355942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:10.355862    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"]
Nov 02 15:20:10.441177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:10.441146    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:10.601710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:10.601565    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:10.791297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:10.791262    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:20:10.869486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:10.869449    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy" containerID="cri-o://72ef6e99d8b71199f37a1d5ea684667214857f35ead9d0543d281ce54cd3d393" gracePeriod=30
Nov 02 15:20:11.128215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.128170    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"]
Nov 02 15:20:11.132955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.132930    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.160461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.160433    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:20:11.160893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.160842    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:20:11.161607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.161582    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"]
Nov 02 15:20:11.224592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.224556    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.224905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.224835    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.225009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.224986    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgrd\" (UniqueName: \"kubernetes.io/projected/95ede21b-827c-4ee0-a93f-4f20175dc1d3-kube-api-access-drgrd\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.326943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.326907    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgrd\" (UniqueName: \"kubernetes.io/projected/95ede21b-827c-4ee0-a93f-4f20175dc1d3-kube-api-access-drgrd\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.327125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.327101    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.327291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.327272    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.327965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.327938    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.329515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.329491    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.353387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.353360    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgrd\" (UniqueName: \"kubernetes.io/projected/95ede21b-827c-4ee0-a93f-4f20175dc1d3-kube-api-access-drgrd\") pod \"maas-api-revoke-keys-e2e-api-b-5f9166-l95cb\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.447814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.447723    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:11.764230 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:20:11.764199    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ede21b_827c_4ee0_a93f_4f20175dc1d3.slice/crio-05f5ff4eaf7a9a3be8e999ea22c244436d1f724cca2b67cfa3ec4fa6ac0386b0 WatchSource:0}: Error finding container 05f5ff4eaf7a9a3be8e999ea22c244436d1f724cca2b67cfa3ec4fa6ac0386b0: Status 404 returned error can't find the container with id 05f5ff4eaf7a9a3be8e999ea22c244436d1f724cca2b67cfa3ec4fa6ac0386b0
Nov 02 15:20:11.768791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.768764    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"]
Nov 02 15:20:11.881235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:11.881196    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb" event={"ID":"95ede21b-827c-4ee0-a93f-4f20175dc1d3","Type":"ContainerStarted","Data":"05f5ff4eaf7a9a3be8e999ea22c244436d1f724cca2b67cfa3ec4fa6ac0386b0"}
Nov 02 15:20:12.773001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:12.772966    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:12.891694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:12.891609    7019 generic.go:356] "Generic (PLEG): container finished" podID="95ede21b-827c-4ee0-a93f-4f20175dc1d3" containerID="0fdbc1c72910226dc378394705068e30d6ec9132b7c2b9d1b3b8e58bf0e9fa02" exitCode=0
Nov 02 15:20:12.891694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:12.891680    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb" event={"ID":"95ede21b-827c-4ee0-a93f-4f20175dc1d3","Type":"ContainerDied","Data":"0fdbc1c72910226dc378394705068e30d6ec9132b7c2b9d1b3b8e58bf0e9fa02"}
Nov 02 15:20:14.297609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.297583    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:14.373091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.373055    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-openshift-ca\") pod \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") "
Nov 02 15:20:14.373289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.373123    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ede21b-827c-4ee0-a93f-4f20175dc1d3-kube-api-access-drgrd\" (UniqueName: \"kubernetes.io/projected/95ede21b-827c-4ee0-a93f-4f20175dc1d3-kube-api-access-drgrd\") pod \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") "
Nov 02 15:20:14.373289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.373263    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-xks-ca\") pod \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\" (UID: \"95ede21b-827c-4ee0-a93f-4f20175dc1d3\") "
Nov 02 15:20:14.373734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.373517    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-openshift-ca" pod "95ede21b-827c-4ee0-a93f-4f20175dc1d3" (UID: "95ede21b-827c-4ee0-a93f-4f20175dc1d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:14.374099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.374073    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:14.375673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.375632    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ede21b-827c-4ee0-a93f-4f20175dc1d3-kube-api-access-drgrd" pod "95ede21b-827c-4ee0-a93f-4f20175dc1d3" (UID: "95ede21b-827c-4ee0-a93f-4f20175dc1d3"). InnerVolumeSpecName "kube-api-access-drgrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:14.375772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.375628    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-xks-ca" pod "95ede21b-827c-4ee0-a93f-4f20175dc1d3" (UID: "95ede21b-827c-4ee0-a93f-4f20175dc1d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:14.475727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.475682    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drgrd\" (UniqueName: \"kubernetes.io/projected/95ede21b-827c-4ee0-a93f-4f20175dc1d3-kube-api-access-drgrd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:14.475727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.475717    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ede21b-827c-4ee0-a93f-4f20175dc1d3-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:14.689083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.689042    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:14.908670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.902173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:14.967729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.952366    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb" event={"ID":"95ede21b-827c-4ee0-a93f-4f20175dc1d3","Type":"ContainerDied","Data":"05f5ff4eaf7a9a3be8e999ea22c244436d1f724cca2b67cfa3ec4fa6ac0386b0"}
Nov 02 15:20:14.967729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.952433    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f5ff4eaf7a9a3be8e999ea22c244436d1f724cca2b67cfa3ec4fa6ac0386b0"
Nov 02 15:20:14.967729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:14.952508    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"
Nov 02 15:20:15.089376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:15.089343    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:16.140445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.140406    7019 generic.go:356] "Generic (PLEG): container finished" podID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerID="72ef6e99d8b71199f37a1d5ea684667214857f35ead9d0543d281ce54cd3d393" exitCode=0
Nov 02 15:20:16.141526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.141480    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" event={"ID":"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0","Type":"ContainerDied","Data":"72ef6e99d8b71199f37a1d5ea684667214857f35ead9d0543d281ce54cd3d393"}
Nov 02 15:20:16.643233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.643204    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:20:16.695670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.695394    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istiod-ca-cert\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.695670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.695472    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-certs\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.695670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.695561    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-socket\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.695670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.695598    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-kube-api-access-7w5zn\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-kube-api-access-7w5zn\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.696069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.695641    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-podinfo\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.696069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.695733    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-envoy\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.696185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.696130    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-data\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.696386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.696363    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-token\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-token\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.696470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.696378    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-certs" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.696470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.696404    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-credential-socket\") pod \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\" (UID: \"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0\") "
Nov 02 15:20:16.696581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.696518    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-socket" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.697043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.696754    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-data" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.697043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.697029    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-credential-socket" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.697528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.697500    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istiod-ca-cert" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:16.697641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.697614    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.697728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.697672    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.697728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.697688    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.697728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.697701    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.699118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.698922    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-kube-api-access-7w5zn" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "kube-api-access-7w5zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:16.699118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.698952    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-token" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:16.699302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.699161    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-envoy" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:20:16.699748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.699724    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-podinfo" pod "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" (UID: "3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:20:16.719912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.719876    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:16.812675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.807736    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.812675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.807765    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.812675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.807778    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5zn\" (UniqueName: \"kubernetes.io/projected/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-kube-api-access-7w5zn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.812675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.807792    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:16.812675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:16.807805    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:20:17.151272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:17.151240    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"
Nov 02 15:20:17.151796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:17.151265    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj" event={"ID":"3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0","Type":"ContainerDied","Data":"9e8cf5aff3560efd87f0b540d412280956c44447cdb34f54181aaaade5362993"}
Nov 02 15:20:17.151796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:17.151332    7019 scope.go:122] "RemoveContainer" containerID="72ef6e99d8b71199f37a1d5ea684667214857f35ead9d0543d281ce54cd3d393"
Nov 02 15:20:17.244087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:17.244055    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"]
Nov 02 15:20:17.247851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:17.247823    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ad276f-openshift-default-7cb6d54d77-fjpsj"]
Nov 02 15:20:17.904697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:17.897457    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" path="/var/lib/kubelet/pods/3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0/volumes"
Nov 02 15:20:17.917318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:17.917276    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:18.318211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.318177    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"]
Nov 02 15:20:18.321876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.320443    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy"
Nov 02 15:20:18.321876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.320469    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy"
Nov 02 15:20:18.321876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.320527    7019 cpu_manager.go:414] "containerMap: removing container" podUID="95ede21b-827c-4ee0-a93f-4f20175dc1d3" containerName="revoke-keys"
Nov 02 15:20:18.321876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.320534    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ede21b-827c-4ee0-a93f-4f20175dc1d3" containerName="revoke-keys"
Nov 02 15:20:18.321876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.320711    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edb6e4b-d183-4ae9-9f47-7c68c8dce8c0" containerName="istio-proxy"
Nov 02 15:20:18.321876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.320728    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ede21b-827c-4ee0-a93f-4f20175dc1d3" containerName="revoke-keys"
Nov 02 15:20:18.327316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.327291    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.358235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.358200    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"]
Nov 02 15:20:18.360688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.360358    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1cdbe6-openshift-default-dockercfg-rv5n9\""
Nov 02 15:20:18.437510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.437477    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:18.458277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.458230    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-token\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.458673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.458621    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhqbd\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-kube-api-access-dhqbd\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.458846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.458827    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-envoy\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.463852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.463825    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-data\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.464129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.464109    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-podinfo\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.464365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.464345    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-certs\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.464796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.464772    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istiod-ca-cert\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.468808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.468783    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-socket\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.469142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.469123    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-credential-socket\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.575976    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-credential-socket\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576090    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-token\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576195    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhqbd\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-kube-api-access-dhqbd\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576222    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-envoy\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576327    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-data\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576374    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-podinfo\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576416    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-certs\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576441    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istiod-ca-cert\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.576509    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-socket\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.577031    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-socket\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.579595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.577319    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-credential-socket\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.641564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.618493    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-podinfo\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.641564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.618809    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-data\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.662673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.648074    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-certs\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.703672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.699907    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istiod-ca-cert\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.707454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.704228    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:18.753738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.750230    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-envoy\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.795692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.789295    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-token\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.795692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.790002    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhqbd\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-kube-api-access-dhqbd\") pod \"e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") " pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:18.868689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.868639    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:18.955333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:18.954290    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:19.416817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:19.416068    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:19.416817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:19.416154    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:19.416817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:19.416197    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:19.418357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:19.418329    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"]
Nov 02 15:20:20.226612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:20.226573    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" event={"ID":"7914308d-0646-4ff8-a269-63cfe6b0b2bb","Type":"ContainerStarted","Data":"05cc727b55d2f1e9a25cdfc42deae71e2126c5370db688f13b48c645964625f6"}
Nov 02 15:20:20.226837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:20.226629    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" event={"ID":"7914308d-0646-4ff8-a269-63cfe6b0b2bb","Type":"ContainerStarted","Data":"68d2459a2643f2a6b9fdfc1bad5a686891ec0e37aaa51fb70595db15203b6cd0"}
Nov 02 15:20:20.295673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:20.294845    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podStartSLOduration=2.294823976 podStartE2EDuration="2.294823976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:20.293143474 +0000 UTC m=+1414.547097098" watchObservedRunningTime="2026-11-02 15:20:20.294823976 +0000 UTC m=+1414.548777602"
Nov 02 15:20:20.316692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:20.316295    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:20.954954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:20.954913    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:20.961160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:20.961134    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:21.234210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:21.234122    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:21.235438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:21.235415    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:20:21.810982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:21.810946    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:22.116314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:22.116279    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:22.261255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:22.261217    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:22.708598 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:20:22.708531    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:20:22.871472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:22.871432    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:23.076815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:23.076617    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:24.284667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:24.282713    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:25.909112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:25.909075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:26.215790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:26.215703    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:26.705762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:26.705209    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": read tcp 10.134.0.2:51890->10.134.0.102:15021: read: connection reset by peer" start-of-body=
Nov 02 15:20:26.705762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:26.705291    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": read tcp 10.134.0.2:51890->10.134.0.102:15021: read: connection reset by peer"
Nov 02 15:20:27.320962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:27.320920    7019 generic.go:356] "Generic (PLEG): container finished" podID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerID="ba743bb9b6c91e91893ff6ac1bf55fa01653c4c533cc64af65bb7f1a46219112" exitCode=0
Nov 02 15:20:27.321477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:27.321033    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" event={"ID":"e7c27a52-60bb-46a5-b431-0d231bfaf36a","Type":"ContainerDied","Data":"ba743bb9b6c91e91893ff6ac1bf55fa01653c4c533cc64af65bb7f1a46219112"}
Nov 02 15:20:27.322024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:27.322004    7019 scope.go:122] "RemoveContainer" containerID="ba743bb9b6c91e91893ff6ac1bf55fa01653c4c533cc64af65bb7f1a46219112"
Nov 02 15:20:27.324210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:27.324173    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:27.324321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:27.324264    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:27.324321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:27.324310    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:20:27.683693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:27.681340    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:28.331670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.319493    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:28.349538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.349502    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" event={"ID":"e7c27a52-60bb-46a5-b431-0d231bfaf36a","Type":"ContainerStarted","Data":"0f0a54b7c2fa8eeae09e66a574b0763f44dc35b7a1a7145cef4fb498cd6893f8"}
Nov 02 15:20:28.488688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.480529    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:20:28.488688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.481028    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 15:20:28.488688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.481092    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 15:20:28.608063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.607806    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:28.651940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.651903    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"]
Nov 02 15:20:28.658566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.658391    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-gmc7x"]
Nov 02 15:20:28.801672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:28.797805    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:29.483671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:29.483075    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 15:20:29.483671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:29.483155    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 15:20:29.862872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:29.862839    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7420f4a2-e22d-4719-ba7b-59958a3b5ec2" path="/var/lib/kubelet/pods/7420f4a2-e22d-4719-ba7b-59958a3b5ec2/volumes"
Nov 02 15:20:30.496945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:30.495855    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 15:20:30.496945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:30.495943    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 15:20:30.499775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:30.499715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:30.694997 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:30.694957    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:31.057673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:31.047258    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:31.263673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:31.263240    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:31.486672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:31.477205    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 15:20:31.486672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:31.477266    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 15:20:32.423710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:32.410822    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:32.480108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:32.480070    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 15:20:32.480424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:32.480385    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 15:20:32.592413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:32.592365    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:32.960106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:32.960065    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:33.480065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:33.480027    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Nov 02 15:20:33.481876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:33.481844    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Nov 02 15:20:34.385239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:34.385203    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:34.479931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:34.479900    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:20:34.481182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:34.480430    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:20:34.481665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:34.481516    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:20:34.804785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:34.804629    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:35.001075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:35.001028    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:36.260057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:36.260021    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:36.466875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:36.466836    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:36.822963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:36.822931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:38.475176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:38.475140    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:38.654455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:38.654420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:38.859117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:38.859035    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:40.537971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:40.537934    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:40.804062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:40.803969    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:42.661630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:42.661596    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:43.864176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:43.864141    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:44.291916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:44.291719    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:44.563458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:44.563376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:45.053220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:45.053176    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:45.509412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:45.509373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:46.099353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:46.099307    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:46.244010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:46.243974    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:47.859505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:47.859393    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:48.866610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:48.866570    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:49.071462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:49.071416    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:49.741597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:49.741558    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:50.538896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:50.538865    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:50.704081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:50.704044    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:51.728282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:51.728246    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:52.707727 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:20:52.707684    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:20:53.427542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:53.427468    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:55.223982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:55.223946    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:56.785725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:56.785690    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:57.880342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:57.880299    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:58.398060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.398018    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:58.570031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.569990    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:20:58.864266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.864237    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6912-0dc187"]
Nov 02 15:20:58.869610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.869585    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:20:58.888601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.888573    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-0dc187"]
Nov 02 15:20:58.895088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.894447    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dlrv\" (UniqueName: \"kubernetes.io/projected/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05-kube-api-access-8dlrv\") pod \"test-curl-6912-0dc187\" (UID: \"bb046a6a-0e1e-4c82-9af0-9e3d8954ed05\") " pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:20:58.896026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.896003    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcqh6\""
Nov 02 15:20:58.997397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:58.997364    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dlrv\" (UniqueName: \"kubernetes.io/projected/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05-kube-api-access-8dlrv\") pod \"test-curl-6912-0dc187\" (UID: \"bb046a6a-0e1e-4c82-9af0-9e3d8954ed05\") " pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:20:59.023230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:59.023193    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dlrv\" (UniqueName: \"kubernetes.io/projected/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05-kube-api-access-8dlrv\") pod \"test-curl-6912-0dc187\" (UID: \"bb046a6a-0e1e-4c82-9af0-9e3d8954ed05\") " pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:20:59.191312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:59.191213    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:20:59.578165 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:20:59.578118    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb046a6a_0e1e_4c82_9af0_9e3d8954ed05.slice/crio-dadf88f1c2a52adabad6c02cda01a6d07eca5c6bbb01b940a2ac26ef4cb4eb7f WatchSource:0}: Error finding container dadf88f1c2a52adabad6c02cda01a6d07eca5c6bbb01b940a2ac26ef4cb4eb7f: Status 404 returned error can't find the container with id dadf88f1c2a52adabad6c02cda01a6d07eca5c6bbb01b940a2ac26ef4cb4eb7f
Nov 02 15:20:59.579057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:59.579032    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-0dc187"]
Nov 02 15:20:59.800893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:59.800856    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-0dc187" event={"ID":"bb046a6a-0e1e-4c82-9af0-9e3d8954ed05","Type":"ContainerStarted","Data":"48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c"}
Nov 02 15:20:59.801115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:59.800905    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-0dc187" event={"ID":"bb046a6a-0e1e-4c82-9af0-9e3d8954ed05","Type":"ContainerStarted","Data":"dadf88f1c2a52adabad6c02cda01a6d07eca5c6bbb01b940a2ac26ef4cb4eb7f"}
Nov 02 15:20:59.845558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:20:59.845436    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6912-0dc187" podStartSLOduration=1.845421312 podStartE2EDuration="1.845421312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:59.8431387 +0000 UTC m=+1454.097092325" watchObservedRunningTime="2026-11-02 15:20:59.845421312 +0000 UTC m=+1454.099374936"
Nov 02 15:21:00.129831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.129796    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:00.443085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.442962    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:00.809741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.809639    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6912-0dc187"]
Nov 02 15:21:00.820768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.819378    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6912-0dc187"]
Nov 02 15:21:00.963716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.963682    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6912-175d69"]
Nov 02 15:21:00.964459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.964445    7019 cpu_manager.go:414] "containerMap: removing container" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" containerName="test-curl-6912-0dc187"
Nov 02 15:21:00.964505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.964460    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" containerName="test-curl-6912-0dc187"
Nov 02 15:21:00.964580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.964571    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" containerName="test-curl-6912-0dc187"
Nov 02 15:21:00.968948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.968926    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:00.986510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:00.986477    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-175d69"]
Nov 02 15:21:01.055052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.055015    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4jrh\" (UniqueName: \"kubernetes.io/projected/d2171941-7667-40a2-a9fd-2eaddb43228c-kube-api-access-g4jrh\") pod \"test-curl-6912-175d69\" (UID: \"d2171941-7667-40a2-a9fd-2eaddb43228c\") " pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:01.156934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.156881    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4jrh\" (UniqueName: \"kubernetes.io/projected/d2171941-7667-40a2-a9fd-2eaddb43228c-kube-api-access-g4jrh\") pod \"test-curl-6912-175d69\" (UID: \"d2171941-7667-40a2-a9fd-2eaddb43228c\") " pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:01.183247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.183217    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4jrh\" (UniqueName: \"kubernetes.io/projected/d2171941-7667-40a2-a9fd-2eaddb43228c-kube-api-access-g4jrh\") pod \"test-curl-6912-175d69\" (UID: \"d2171941-7667-40a2-a9fd-2eaddb43228c\") " pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:01.284743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.284705    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:01.533736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.529493    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:21:01.740712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.736900    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-175d69"]
Nov 02 15:21:01.740712 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:21:01.737465    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2171941_7667_40a2_a9fd_2eaddb43228c.slice/crio-cd16c509d3aee3fc280ed837fd4b546b91f9a18382f48e9cc23742041e576e5c WatchSource:0}: Error finding container cd16c509d3aee3fc280ed837fd4b546b91f9a18382f48e9cc23742041e576e5c: Status 404 returned error can't find the container with id cd16c509d3aee3fc280ed837fd4b546b91f9a18382f48e9cc23742041e576e5c
Nov 02 15:21:01.835722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.835580    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-175d69" event={"ID":"d2171941-7667-40a2-a9fd-2eaddb43228c","Type":"ContainerStarted","Data":"cd16c509d3aee3fc280ed837fd4b546b91f9a18382f48e9cc23742041e576e5c"}
Nov 02 15:21:01.836064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:01.836037    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6912-0dc187" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" containerName="test-curl-6912-0dc187" containerID="cri-o://48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c" gracePeriod=2
Nov 02 15:21:01.855211 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:01.855166    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-0dc187\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:21:02.054532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:02.053538    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:02.846269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:02.846229    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-175d69" event={"ID":"d2171941-7667-40a2-a9fd-2eaddb43228c","Type":"ContainerStarted","Data":"f1dbf3d5cda0477bc56e60bb5976dc66c58ec95dc1604b0f995e96f3136224f8"}
Nov 02 15:21:03.917671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:03.917092    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6912-175d69" podStartSLOduration=3.9170653189999998 podStartE2EDuration="3.917065319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:02.91189327 +0000 UTC m=+1457.165846895" watchObservedRunningTime="2026-11-02 15:21:03.917065319 +0000 UTC m=+1458.171019013"
Nov 02 15:21:03.927196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:03.927145    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6912-175d69"]
Nov 02 15:21:03.936137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:03.936094    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6912-175d69"]
Nov 02 15:21:04.062428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.062394    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6912-85d8fd"]
Nov 02 15:21:04.063230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.063207    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" containerName="test-curl-6912-175d69"
Nov 02 15:21:04.063230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.063229    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" containerName="test-curl-6912-175d69"
Nov 02 15:21:04.063457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.063433    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" containerName="test-curl-6912-175d69"
Nov 02 15:21:04.133804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.133760    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:04.146050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.146010    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-85d8fd"]
Nov 02 15:21:04.156034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.155997    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w46x\" (UniqueName: \"kubernetes.io/projected/6fffe7c3-06ac-4adf-a717-8d5796e4b85a-kube-api-access-5w46x\") pod \"test-curl-6912-85d8fd\" (UID: \"6fffe7c3-06ac-4adf-a717-8d5796e4b85a\") " pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:04.260054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.260002    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w46x\" (UniqueName: \"kubernetes.io/projected/6fffe7c3-06ac-4adf-a717-8d5796e4b85a-kube-api-access-5w46x\") pod \"test-curl-6912-85d8fd\" (UID: \"6fffe7c3-06ac-4adf-a717-8d5796e4b85a\") " pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:04.360670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.351364    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:04.450922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.415014    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w46x\" (UniqueName: \"kubernetes.io/projected/6fffe7c3-06ac-4adf-a717-8d5796e4b85a-kube-api-access-5w46x\") pod \"test-curl-6912-85d8fd\" (UID: \"6fffe7c3-06ac-4adf-a717-8d5796e4b85a\") " pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:04.460494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.460217    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:04.564200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.562885    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:21:04.685689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.683823    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05-kube-api-access-8dlrv\" (UniqueName: \"kubernetes.io/projected/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05-kube-api-access-8dlrv\") pod \"bb046a6a-0e1e-4c82-9af0-9e3d8954ed05\" (UID: \"bb046a6a-0e1e-4c82-9af0-9e3d8954ed05\") "
Nov 02 15:21:04.713424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.713313    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05-kube-api-access-8dlrv" pod "bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" (UID: "bb046a6a-0e1e-4c82-9af0-9e3d8954ed05"). InnerVolumeSpecName "kube-api-access-8dlrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:04.821051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.821014    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dlrv\" (UniqueName: \"kubernetes.io/projected/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05-kube-api-access-8dlrv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:21:04.944275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.941297    7019 generic.go:356] "Generic (PLEG): container finished" podID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" containerID="48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c" exitCode=137
Nov 02 15:21:04.944275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.941631    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6912-175d69" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" containerName="test-curl-6912-175d69" containerID="cri-o://f1dbf3d5cda0477bc56e60bb5976dc66c58ec95dc1604b0f995e96f3136224f8" gracePeriod=2
Nov 02 15:21:04.944275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.942006    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:21:04.951431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.946791    7019 scope.go:122] "RemoveContainer" containerID="48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c"
Nov 02 15:21:04.952913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:04.952675    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-85d8fd"]
Nov 02 15:21:04.954848 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:21:04.954801    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fffe7c3_06ac_4adf_a717_8d5796e4b85a.slice/crio-aa6047bfb7436ada1ed6bf47c1f5cb4a78f572142cc422641c67c133a9ac1462 WatchSource:0}: Error finding container aa6047bfb7436ada1ed6bf47c1f5cb4a78f572142cc422641c67c133a9ac1462: Status 404 returned error can't find the container with id aa6047bfb7436ada1ed6bf47c1f5cb4a78f572142cc422641c67c133a9ac1462
Nov 02 15:21:04.962107 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:04.962075    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-175d69\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:04.983725 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:04.978475    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-175d69\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:05.054870 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:05.054215    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-175d69\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:05.054870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:05.054454    7019 scope.go:122] "RemoveContainer" containerID="48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c"
Nov 02 15:21:05.055010 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:05.054970    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c\": container with ID starting with 48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c not found: ID does not exist" containerID="48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c"
Nov 02 15:21:05.055064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:05.055012    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c"} err="failed to get container status \"48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c\": rpc error: code = NotFound desc = could not find container \"48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c\": container with ID starting with 48f70ecec361a083830b1cb5d6e72d8e66bf2a3064f88af2af9849d66dcbac4c not found: ID does not exist"
Nov 02 15:21:05.081838 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:05.080386    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-0dc187\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:21:06.326676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.325935    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:06.337686 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:06.329718    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-175d69\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:06.337686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.337224    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" path="/var/lib/kubelet/pods/bb046a6a-0e1e-4c82-9af0-9e3d8954ed05/volumes"
Nov 02 15:21:06.337963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.337734    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"]
Nov 02 15:21:06.351044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.351002    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"]
Nov 02 15:21:06.351388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.351370    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.356221 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:06.356182    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-0dc187\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="bb046a6a-0e1e-4c82-9af0-9e3d8954ed05" pod="opendatahub/test-curl-6912-0dc187"
Nov 02 15:21:06.360909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.360830    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-85d8fd" event={"ID":"6fffe7c3-06ac-4adf-a717-8d5796e4b85a","Type":"ContainerStarted","Data":"2fb7e2e19b5bb2ca66983c4f7c516c852f201383dc02cc92a5202cc92b3a4413"}
Nov 02 15:21:06.361243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.361222    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-85d8fd" event={"ID":"6fffe7c3-06ac-4adf-a717-8d5796e4b85a","Type":"ContainerStarted","Data":"aa6047bfb7436ada1ed6bf47c1f5cb4a78f572142cc422641c67c133a9ac1462"}
Nov 02 15:21:06.373208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.373172    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3d7df16e-serving-cert\""
Nov 02 15:21:06.491688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.489219    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-maas-api-tls\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.491688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.489375    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp6g\" (UniqueName: \"kubernetes.io/projected/ba41140f-790b-4fad-b28b-6d19ce0a9c94-kube-api-access-qnp6g\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.491688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.489423    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-metrics-certs\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.491688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.489687    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba41140f-790b-4fad-b28b-6d19ce0a9c94-openshift-service-ca\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.512684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.512468    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:06.595728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.591922    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba41140f-790b-4fad-b28b-6d19ce0a9c94-openshift-service-ca\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.595728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.592105    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-maas-api-tls\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.595728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.592229    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp6g\" (UniqueName: \"kubernetes.io/projected/ba41140f-790b-4fad-b28b-6d19ce0a9c94-kube-api-access-qnp6g\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.595728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.592272    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-metrics-certs\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.614684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.606482    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-maas-api-tls\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.614992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.608118    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba41140f-790b-4fad-b28b-6d19ce0a9c94-openshift-service-ca\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.615123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.608680    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-metrics-certs\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.630893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.630859    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp6g\" (UniqueName: \"kubernetes.io/projected/ba41140f-790b-4fad-b28b-6d19ce0a9c94-kube-api-access-qnp6g\") pod \"maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:06.712687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:06.697255    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:07.423684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.422493    7019 generic.go:356] "Generic (PLEG): container finished" podID="d2171941-7667-40a2-a9fd-2eaddb43228c" containerID="f1dbf3d5cda0477bc56e60bb5976dc66c58ec95dc1604b0f995e96f3136224f8" exitCode=137
Nov 02 15:21:07.842341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.824130    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6912-85d8fd" podStartSLOduration=3.824106838 podStartE2EDuration="3.824106838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:06.513730818 +0000 UTC m=+1460.767684443" watchObservedRunningTime="2026-11-02 15:21:07.824106838 +0000 UTC m=+1462.078060464"
Nov 02 15:21:07.842341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.829728    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"]
Nov 02 15:21:07.845498 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:21:07.845464    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba41140f_790b_4fad_b28b_6d19ce0a9c94.slice/crio-16a779c47ef6a2148327b0b041570813ed67659a81132844a9569044d922a434 WatchSource:0}: Error finding container 16a779c47ef6a2148327b0b041570813ed67659a81132844a9569044d922a434: Status 404 returned error can't find the container with id 16a779c47ef6a2148327b0b041570813ed67659a81132844a9569044d922a434
Nov 02 15:21:07.945720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.945687    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:07.956310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.956267    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2171941-7667-40a2-a9fd-2eaddb43228c-kube-api-access-g4jrh\" (UniqueName: \"kubernetes.io/projected/d2171941-7667-40a2-a9fd-2eaddb43228c-kube-api-access-g4jrh\") pod \"d2171941-7667-40a2-a9fd-2eaddb43228c\" (UID: \"d2171941-7667-40a2-a9fd-2eaddb43228c\") "
Nov 02 15:21:07.981021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.980042    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2171941-7667-40a2-a9fd-2eaddb43228c-kube-api-access-g4jrh" pod "d2171941-7667-40a2-a9fd-2eaddb43228c" (UID: "d2171941-7667-40a2-a9fd-2eaddb43228c"). InnerVolumeSpecName "kube-api-access-g4jrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:07.981021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.962319    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:08.017676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:07.998937    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4jrh\" (UniqueName: \"kubernetes.io/projected/d2171941-7667-40a2-a9fd-2eaddb43228c-kube-api-access-g4jrh\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:21:08.345769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.345688    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6912-85d8fd"]
Nov 02 15:21:08.424925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.382204    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6912-85d8fd"]
Nov 02 15:21:08.454016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.453873    7019 scope.go:122] "RemoveContainer" containerID="f1dbf3d5cda0477bc56e60bb5976dc66c58ec95dc1604b0f995e96f3136224f8"
Nov 02 15:21:08.454016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.453876    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:08.461932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.460837    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6912-85d8fd" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" containerName="test-curl-6912-85d8fd" containerID="cri-o://2fb7e2e19b5bb2ca66983c4f7c516c852f201383dc02cc92a5202cc92b3a4413" gracePeriod=2
Nov 02 15:21:08.461932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.460842    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6" event={"ID":"ba41140f-790b-4fad-b28b-6d19ce0a9c94","Type":"ContainerStarted","Data":"16a779c47ef6a2148327b0b041570813ed67659a81132844a9569044d922a434"}
Nov 02 15:21:08.502205 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:08.486311    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-85d8fd\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:08.541838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.541790    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6912-131856"]
Nov 02 15:21:08.543182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.543154    7019 cpu_manager.go:414] "containerMap: removing container" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" containerName="test-curl-6912-85d8fd"
Nov 02 15:21:08.543182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.543182    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" containerName="test-curl-6912-85d8fd"
Nov 02 15:21:08.545139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.543480    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" containerName="test-curl-6912-85d8fd"
Nov 02 15:21:08.551793 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:08.550563    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-85d8fd\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:08.558090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.558074    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-131856"]
Nov 02 15:21:08.558255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.558245    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:08.598816 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:08.596920    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-175d69\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:08.614673 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:08.613901    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-175d69\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:08.649881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.644556    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsnd7\" (UniqueName: \"kubernetes.io/projected/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9-kube-api-access-qsnd7\") pod \"test-curl-6912-131856\" (UID: \"f7f736a1-d1f4-44d8-bbcd-f83511feb7e9\") " pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:08.705685 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:08.682038    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-85d8fd\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:08.745901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.745851    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsnd7\" (UniqueName: \"kubernetes.io/projected/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9-kube-api-access-qsnd7\") pod \"test-curl-6912-131856\" (UID: \"f7f736a1-d1f4-44d8-bbcd-f83511feb7e9\") " pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:08.822730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.819268    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsnd7\" (UniqueName: \"kubernetes.io/projected/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9-kube-api-access-qsnd7\") pod \"test-curl-6912-131856\" (UID: \"f7f736a1-d1f4-44d8-bbcd-f83511feb7e9\") " pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:08.944229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:08.944191    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:09.341592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:09.341409    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6912-131856"]
Nov 02 15:21:09.343053 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:21:09.343025    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7f736a1_d1f4_44d8_bbcd_f83511feb7e9.slice/crio-455d5d0160813012d6fd146d7505d606d04484cc96b2d0d6024f9627d34aa466 WatchSource:0}: Error finding container 455d5d0160813012d6fd146d7505d606d04484cc96b2d0d6024f9627d34aa466: Status 404 returned error can't find the container with id 455d5d0160813012d6fd146d7505d606d04484cc96b2d0d6024f9627d34aa466
Nov 02 15:21:09.523682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:09.523442    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6" event={"ID":"ba41140f-790b-4fad-b28b-6d19ce0a9c94","Type":"ContainerStarted","Data":"c8f58fcd1ece4d066ac97ccf2b25add306f70a6aeee9fe0b6bd0b60e02c14a18"}
Nov 02 15:21:09.534661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:09.524818    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:09.534661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:09.529485    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-131856" event={"ID":"f7f736a1-d1f4-44d8-bbcd-f83511feb7e9","Type":"ContainerStarted","Data":"455d5d0160813012d6fd146d7505d606d04484cc96b2d0d6024f9627d34aa466"}
Nov 02 15:21:09.539693 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:09.539548    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-175d69\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" pod="opendatahub/test-curl-6912-175d69"
Nov 02 15:21:09.554703 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:09.554577    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-85d8fd\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:09.633553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:09.633489    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6" podStartSLOduration=4.22751899 podStartE2EDuration="4.633470145s" totalImagesPullingTime="405.951155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:09.630802111 +0000 UTC m=+1463.884755735" watchObservedRunningTime="2026-11-02 15:21:09.633470145 +0000 UTC m=+1463.887423816"
Nov 02 15:21:09.744234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:09.744147    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:09.867779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:09.867046    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2171941-7667-40a2-a9fd-2eaddb43228c" path="/var/lib/kubelet/pods/d2171941-7667-40a2-a9fd-2eaddb43228c/volumes"
Nov 02 15:21:10.539129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:10.539092    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6912-131856" event={"ID":"f7f736a1-d1f4-44d8-bbcd-f83511feb7e9","Type":"ContainerStarted","Data":"c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f"}
Nov 02 15:21:10.542716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:10.542674    7019 generic.go:356] "Generic (PLEG): container finished" podID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" containerID="2fb7e2e19b5bb2ca66983c4f7c516c852f201383dc02cc92a5202cc92b3a4413" exitCode=137
Nov 02 15:21:10.598586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:10.598514    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6912-131856" podStartSLOduration=2.598492395 podStartE2EDuration="2.598492395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:10.581184171 +0000 UTC m=+1464.835137800" watchObservedRunningTime="2026-11-02 15:21:10.598492395 +0000 UTC m=+1464.852446020"
Nov 02 15:21:10.690266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:10.690227    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:10.793442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:10.793339    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fffe7c3-06ac-4adf-a717-8d5796e4b85a-kube-api-access-5w46x\" (UniqueName: \"kubernetes.io/projected/6fffe7c3-06ac-4adf-a717-8d5796e4b85a-kube-api-access-5w46x\") pod \"6fffe7c3-06ac-4adf-a717-8d5796e4b85a\" (UID: \"6fffe7c3-06ac-4adf-a717-8d5796e4b85a\") "
Nov 02 15:21:10.795930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:10.795871    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fffe7c3-06ac-4adf-a717-8d5796e4b85a-kube-api-access-5w46x" pod "6fffe7c3-06ac-4adf-a717-8d5796e4b85a" (UID: "6fffe7c3-06ac-4adf-a717-8d5796e4b85a"). InnerVolumeSpecName "kube-api-access-5w46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:10.897105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:10.897066    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w46x\" (UniqueName: \"kubernetes.io/projected/6fffe7c3-06ac-4adf-a717-8d5796e4b85a-kube-api-access-5w46x\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:21:11.358802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:11.358761    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:11.511671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:11.511308    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6912-131856"]
Nov 02 15:21:11.519938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:11.519903    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6912-131856"]
Nov 02 15:21:11.590711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:11.587101    7019 scope.go:122] "RemoveContainer" containerID="2fb7e2e19b5bb2ca66983c4f7c516c852f201383dc02cc92a5202cc92b3a4413"
Nov 02 15:21:11.590711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:11.587145    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:11.619784 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:11.618199    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-85d8fd\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" pod="opendatahub/test-curl-6912-85d8fd"
Nov 02 15:21:11.866394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:11.866348    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fffe7c3-06ac-4adf-a717-8d5796e4b85a" path="/var/lib/kubelet/pods/6fffe7c3-06ac-4adf-a717-8d5796e4b85a/volumes"
Nov 02 15:21:12.154455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.154415    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:12.501081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.497918    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"]
Nov 02 15:21:12.501081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.499284    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" containerName="test-curl-6912-131856"
Nov 02 15:21:12.501081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.499310    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" containerName="test-curl-6912-131856"
Nov 02 15:21:12.501081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.499492    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" containerName="test-curl-6912-131856"
Nov 02 15:21:12.530334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.514691    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.541462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.538348    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f975ed-openshift-default-dockercfg-xjt5f\""
Nov 02 15:21:12.565672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.560442    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"]
Nov 02 15:21:12.618672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.604515    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:12.618672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.611526    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6912-131856" podUID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" containerName="test-curl-6912-131856" containerID="cri-o://c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f" gracePeriod=2
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641134    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zfb2\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-kube-api-access-2zfb2\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641415    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-data\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641504    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-podinfo\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641588    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-token\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641627    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-credential-socket\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641667    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-envoy\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641714    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-socket\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641772    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-certs\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.645787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.641814    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istiod-ca-cert\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.750114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.749739    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-token\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.750114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.749800    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-credential-socket\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.750114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.749827    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-envoy\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.750114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.749870    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-socket\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.750114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.749928    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-certs\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.750114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.749970    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istiod-ca-cert\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.750114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.750021    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zfb2\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-kube-api-access-2zfb2\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.776612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.769326    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-data\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.776612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.769467    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-podinfo\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.796937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.791029    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-credential-socket\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.798968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.798062    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istiod-ca-cert\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.798968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.798336    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-socket\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.798968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.798563    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-certs\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.798968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.798849    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-data\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.824596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.819440    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:12.862077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.862006    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zfb2\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-kube-api-access-2zfb2\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.864917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.864853    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-envoy\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.872462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.872398    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-token\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:12.873265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:12.873238    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-podinfo\") pod \"e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") " pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:13.182676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:13.172964    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:13.870767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:13.870682    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:13.874839 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:21:13.874783    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377eabb1_c526_4a6b_9d85_05d6004a0cf9.slice/crio-73f040a5953e081abd4b90a3af55ee51a36e3510bcdbed1c5dc211cf22ba2761 WatchSource:0}: Error finding container 73f040a5953e081abd4b90a3af55ee51a36e3510bcdbed1c5dc211cf22ba2761: Status 404 returned error can't find the container with id 73f040a5953e081abd4b90a3af55ee51a36e3510bcdbed1c5dc211cf22ba2761
Nov 02 15:21:13.886595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:13.886473    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:13.886595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:13.886581    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:13.886873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:13.886630    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:13.893227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:13.893128    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"]
Nov 02 15:21:14.047814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:14.047727    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:14.659429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:14.658075    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" event={"ID":"377eabb1-c526-4a6b-9d85-05d6004a0cf9","Type":"ContainerStarted","Data":"775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf"}
Nov 02 15:21:14.659429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:14.658153    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" event={"ID":"377eabb1-c526-4a6b-9d85-05d6004a0cf9","Type":"ContainerStarted","Data":"73f040a5953e081abd4b90a3af55ee51a36e3510bcdbed1c5dc211cf22ba2761"}
Nov 02 15:21:14.742707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:14.742635    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" podStartSLOduration=2.742617351 podStartE2EDuration="2.742617351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:14.735893539 +0000 UTC m=+1468.989847163" watchObservedRunningTime="2026-11-02 15:21:14.742617351 +0000 UTC m=+1468.996570974"
Nov 02 15:21:15.183994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.174729    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:15.192634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.191696    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:15.469011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.468916    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:15.534018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.533983    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9-kube-api-access-qsnd7\" (UniqueName: \"kubernetes.io/projected/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9-kube-api-access-qsnd7\") pod \"f7f736a1-d1f4-44d8-bbcd-f83511feb7e9\" (UID: \"f7f736a1-d1f4-44d8-bbcd-f83511feb7e9\") "
Nov 02 15:21:15.536347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.536315    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9-kube-api-access-qsnd7" pod "f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" (UID: "f7f736a1-d1f4-44d8-bbcd-f83511feb7e9"). InnerVolumeSpecName "kube-api-access-qsnd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:15.636080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.636042    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsnd7\" (UniqueName: \"kubernetes.io/projected/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9-kube-api-access-qsnd7\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.664734    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.684226    7019 generic.go:356] "Generic (PLEG): container finished" podID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" containerID="c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f" exitCode=137
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.684669    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.698791    7019 scope.go:122] "RemoveContainer" containerID="c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f"
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.700499    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.710621    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.728499    7019 scope.go:122] "RemoveContainer" containerID="c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f"
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:15.729283    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f\": container with ID starting with c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f not found: ID does not exist" containerID="c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f"
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.729325    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f"} err="failed to get container status \"c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f\": rpc error: code = NotFound desc = could not find container \"c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f\": container with ID starting with c47013dd3f602b4bc50a8c70db3f4ee9163845a362dcb1afe468a5532faa969f not found: ID does not exist"
Nov 02 15:21:15.790006 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:15.777355    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-131856\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:15.896845 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:15.883506    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6912-131856\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" pod="opendatahub/test-curl-6912-131856"
Nov 02 15:21:15.904065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:15.904030    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f736a1-d1f4-44d8-bbcd-f83511feb7e9" path="/var/lib/kubelet/pods/f7f736a1-d1f4-44d8-bbcd-f83511feb7e9/volumes"
Nov 02 15:21:16.595841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.595810    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:21:16.702637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.702604    7019 generic.go:356] "Generic (PLEG): container finished" podID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerID="fddb1e4877332ad6835a7f515ef91326f53adf852fc83bb763a55975c5bc6077" exitCode=0
Nov 02 15:21:16.702931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.702712    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" event={"ID":"62dde604-430f-4f5f-bae4-111c194a1d2b","Type":"ContainerDied","Data":"fddb1e4877332ad6835a7f515ef91326f53adf852fc83bb763a55975c5bc6077"}
Nov 02 15:21:16.703377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.703351    7019 scope.go:122] "RemoveContainer" containerID="fddb1e4877332ad6835a7f515ef91326f53adf852fc83bb763a55975c5bc6077"
Nov 02 15:21:16.706459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.706423    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:16.706604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.706518    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:16.706743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.706601    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:16.707085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.707058    7019 generic.go:356] "Generic (PLEG): container finished" podID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerID="d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87" exitCode=0
Nov 02 15:21:16.707642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.707149    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" event={"ID":"4bf5e56a-3578-433f-9169-a1901aeb6de6","Type":"ContainerDied","Data":"d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87"}
Nov 02 15:21:16.707999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.707974    7019 scope.go:122] "RemoveContainer" containerID="d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87"
Nov 02 15:21:16.710451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.710417    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:16.710567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.710508    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:16.710636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:16.710611    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:17.382274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:17.380506    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:17.586697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:17.586277    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:17.796748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:17.790403    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" event={"ID":"62dde604-430f-4f5f-bae4-111c194a1d2b","Type":"ContainerStarted","Data":"918f385c146a1bfcb370062b8d8889cd90d8ca89d9d5767dea7c0445f657d405"}
Nov 02 15:21:17.810674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:17.801998    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" event={"ID":"4bf5e56a-3578-433f-9169-a1901aeb6de6","Type":"ContainerStarted","Data":"dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f"}
Nov 02 15:21:18.051724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:18.051210    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:21:18.052794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:18.052758    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:21:18.052982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:18.052961    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 15:21:18.068994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:18.066106    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:18.328236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:18.328143    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:21:18.328606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:18.328581    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:21:18.328792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:18.328766    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 15:21:19.051905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:19.051874    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:21:19.052333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:19.051935    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 15:21:19.248968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:19.248928    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:19.333798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:19.327372    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:21:19.333798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:19.327499    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 15:21:19.466673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:19.465899    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:19.677672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:19.650504    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:20.052184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:20.052092    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:21:20.052184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:20.052152    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 15:21:20.329813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:20.329712    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:21:20.329813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:20.329792    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 15:21:21.055728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:21.052464    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:21:21.055728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:21.052534    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 15:21:21.330731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:21.330597    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:21:21.330731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:21.330679    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 15:21:21.536670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:21.531865    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:22.051755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:22.051714    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:21:22.051934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:22.051770    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 15:21:22.330784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:22.327915    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:21:22.330784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:22.327981    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 15:21:22.753726 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:22.753690    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:21:23.051805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:23.051721    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Nov 02 15:21:23.051805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:23.051773    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Nov 02 15:21:23.243415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:23.243375    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:23.332737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:23.332455    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 02 15:21:23.332737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:23.332518    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 02 15:21:24.056613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:24.056583    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:21:24.057016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:24.056997    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:21:24.057744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:24.057727    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:21:24.330665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:24.330572    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:21:24.331024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:24.331005    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:21:24.331850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:24.331831    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:21:25.015562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:25.015516    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:26.634246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:26.634206    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:28.159322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:28.159282    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:28.313152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:28.313115    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:28.465189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:28.465094    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:29.913547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:29.913508    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:30.116570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.116532    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"]
Nov 02 15:21:30.129845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.129813    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.135498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.135463    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"]
Nov 02 15:21:30.148573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.148552    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a713aa\"/\"kube-root-ca.crt\""
Nov 02 15:21:30.161999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.161963    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a713aa\"/\"openshift-service-ca.crt\""
Nov 02 15:21:30.162161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.162075    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a713aa\"/\"test-model-a713aa-kserve-self-signed-certs\""
Nov 02 15:21:30.162161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.162088    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a713aa\"/\"default-dockercfg-5vt2t\""
Nov 02 15:21:30.205377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.205293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:30.212008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.211977    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgd9\" (UniqueName: \"kubernetes.io/projected/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-kube-api-access-gqgd9\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.212166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.212126    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-home\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.212259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.212159    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-model-cache\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.212259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.212197    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-dshm\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.212259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.212221    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tls-certs\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.212420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.212264    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tmp-dir\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.313765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.313716    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgd9\" (UniqueName: \"kubernetes.io/projected/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-kube-api-access-gqgd9\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.313965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.313911    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-home\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.313965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.313940    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-model-cache\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.314095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.313971    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-dshm\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.314095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.314012    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tls-certs\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.314095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.314064    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tmp-dir\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.314412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.314385    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-model-cache\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.314538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.314385    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-home\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.314701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.314637    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tmp-dir\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.317216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.317190    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-dshm\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.317479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.317461    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tls-certs\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.339324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.339287    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgd9\" (UniqueName: \"kubernetes.io/projected/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-kube-api-access-gqgd9\") pod \"test-model-a713aa-kserve-66d6c565bc-p7bkw\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") " pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.457717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.449503    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:30.925943 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:21:30.925913    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b6ac155_b878_4a2e_a9a5_7e3a1c2d1d4e.slice/crio-7dd6d3c228df09dcc1903f6cb5e39d404507ef601a3d17860ff6dc019cd80d41 WatchSource:0}: Error finding container 7dd6d3c228df09dcc1903f6cb5e39d404507ef601a3d17860ff6dc019cd80d41: Status 404 returned error can't find the container with id 7dd6d3c228df09dcc1903f6cb5e39d404507ef601a3d17860ff6dc019cd80d41
Nov 02 15:21:30.926434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:30.926412    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"]
Nov 02 15:21:31.149801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:31.149744    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw" event={"ID":"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e","Type":"ContainerStarted","Data":"a4e65c781531684acd400c857df25c062be0d1cc514d107205fd2fd6567a39a7"}
Nov 02 15:21:31.150014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:31.149832    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw" event={"ID":"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e","Type":"ContainerStarted","Data":"7dd6d3c228df09dcc1903f6cb5e39d404507ef601a3d17860ff6dc019cd80d41"}
Nov 02 15:21:31.197544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:31.197422    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw" podStartSLOduration=1.197407231 podStartE2EDuration="1.197407231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:31.194922486 +0000 UTC m=+1485.448876111" watchObservedRunningTime="2026-11-02 15:21:31.197407231 +0000 UTC m=+1485.451360854"
Nov 02 15:21:31.452695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:31.452575    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:31.709369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:31.709264    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:33.212143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:33.212106    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:33.899050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:33.873854    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:34.209912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:34.209824    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:34.257282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:34.257248    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"]
Nov 02 15:21:34.270721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:34.270692    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5f9166-l95cb"]
Nov 02 15:21:34.413677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:34.410674    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:34.669062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:34.669010    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:35.863868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:35.863835    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ede21b-827c-4ee0-a93f-4f20175dc1d3" path="/var/lib/kubelet/pods/95ede21b-827c-4ee0-a93f-4f20175dc1d3/volumes"
Nov 02 15:21:36.157404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:36.157261    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:37.877785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:37.877746    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:39.573959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:39.573924    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:40.054153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:40.054118    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"]
Nov 02 15:21:40.069589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:40.069558    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c8883b2-q5khd"]
Nov 02 15:21:40.452450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:40.452412    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:40.465617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:40.465586    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:41.256470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:41.256436    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:41.287716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:41.287681    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:21:41.424255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:41.424180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:41.620443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:41.620402    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:41.864539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:41.864507    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2add2e6e-c0bf-4799-a999-7f036a67eaac" path="/var/lib/kubelet/pods/2add2e6e-c0bf-4799-a999-7f036a67eaac/volumes"
Nov 02 15:21:41.888466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:41.888382    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:42.592049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:42.592016    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:43.152798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:43.151018    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:43.554693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:43.554580    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:43.814249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:43.813362    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:44.568998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:44.568957    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:45.391441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:45.391401    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:46.212351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:46.209515    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:46.610793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:46.592397    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:46.788576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:46.788533    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:46.980728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:46.974811    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:47.237736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:47.236022    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:47.787608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:47.787576    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:47.946804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:47.946745    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:49.229719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:49.229684    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:49.697364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:49.697327    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:49.852334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:49.852298    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:50.932761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:50.932727    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:51.118003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:51.112840    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:51.263886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:51.263780    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:52.714017 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:21:52.713970    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:21:52.736554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.736521    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:52.836827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.836794    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"]
Nov 02 15:21:52.842966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.842934    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:52.862689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.862617    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"]
Nov 02 15:21:52.869581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.869556    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4ec02d\""
Nov 02 15:21:52.916276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.916237    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:52.948469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.948427    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6sjm\" (UniqueName: \"kubernetes.io/projected/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-kube-api-access-b6sjm\") pod \"payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:52.948701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:52.948674    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:53.148672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.145204    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:53.148672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.145469    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6sjm\" (UniqueName: \"kubernetes.io/projected/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-kube-api-access-b6sjm\") pod \"payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:53.148672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.146626    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:53.231902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.221856    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:53.231902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.227583    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6sjm\" (UniqueName: \"kubernetes.io/projected/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-kube-api-access-b6sjm\") pod \"payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:53.379548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.379509    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:53.462172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.462087    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:53.804843 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:21:53.804812    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca81c9f4_1de7_4bfa_abfe_7a05d5839b15.slice/crio-d1648ffa518091cfbe0165851f53670fbc99fae41f6e0150fdf5e88694ac0779 WatchSource:0}: Error finding container d1648ffa518091cfbe0165851f53670fbc99fae41f6e0150fdf5e88694ac0779: Status 404 returned error can't find the container with id d1648ffa518091cfbe0165851f53670fbc99fae41f6e0150fdf5e88694ac0779
Nov 02 15:21:53.806630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:53.806606    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"]
Nov 02 15:21:54.798973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:54.798935    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq" event={"ID":"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15","Type":"ContainerStarted","Data":"fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45"}
Nov 02 15:21:54.799197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:54.799050    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq" event={"ID":"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15","Type":"ContainerStarted","Data":"d1648ffa518091cfbe0165851f53670fbc99fae41f6e0150fdf5e88694ac0779"}
Nov 02 15:21:54.799197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:54.799096    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:21:54.845689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:54.845591    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq" podStartSLOduration=2.845575589 podStartE2EDuration="2.845575589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:54.843846195 +0000 UTC m=+1509.097799821" watchObservedRunningTime="2026-11-02 15:21:54.845575589 +0000 UTC m=+1509.099529262"
Nov 02 15:21:55.325676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:55.322701    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:56.986982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:56.986886    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:57.825177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:57.825146    7019 generic.go:356] "Generic (PLEG): container finished" podID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerID="05cc727b55d2f1e9a25cdfc42deae71e2126c5370db688f13b48c645964625f6" exitCode=0
Nov 02 15:21:57.825443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:57.825224    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" event={"ID":"7914308d-0646-4ff8-a269-63cfe6b0b2bb","Type":"ContainerDied","Data":"05cc727b55d2f1e9a25cdfc42deae71e2126c5370db688f13b48c645964625f6"}
Nov 02 15:21:57.825816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:57.825793    7019 scope.go:122] "RemoveContainer" containerID="05cc727b55d2f1e9a25cdfc42deae71e2126c5370db688f13b48c645964625f6"
Nov 02 15:21:57.827888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:57.827852    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:57.827963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:57.827950    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:57.828008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:57.827984    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:21:58.710544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:58.710492    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:21:58.836767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:58.836735    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" event={"ID":"7914308d-0646-4ff8-a269-63cfe6b0b2bb","Type":"ContainerStarted","Data":"141095b0ceff7217a7759f6aad807129ff8310b7ee858f1103560f1e7c19f5c9"}
Nov 02 15:21:58.955112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:58.955078    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:21:58.955420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:58.955395    7019 patch_prober.go:29] interesting pod/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:21:58.955526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:58.955475    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 15:21:59.955639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:59.955581    7019 patch_prober.go:29] interesting pod/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:21:59.956067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:21:59.955684    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 15:22:00.536450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.536409    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:00.636410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.627345    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6900-af0c54"]
Nov 02 15:22:00.720673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.717630    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:00.726793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.726745    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6900-af0c54"]
Nov 02 15:22:00.726973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.726889    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:00.743977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.743947    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcqh6\""
Nov 02 15:22:00.841594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.841502    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlwl\" (UniqueName: \"kubernetes.io/projected/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5-kube-api-access-2zlwl\") pod \"test-curl-6900-af0c54\" (UID: \"7d63d7ad-0c11-400d-beea-dd7fcb7a86a5\") " pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:00.943690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.943629    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlwl\" (UniqueName: \"kubernetes.io/projected/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5-kube-api-access-2zlwl\") pod \"test-curl-6900-af0c54\" (UID: \"7d63d7ad-0c11-400d-beea-dd7fcb7a86a5\") " pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:00.955812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.955773    7019 patch_prober.go:29] interesting pod/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:22:00.956283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:00.955850    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 15:22:01.005561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.005502    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlwl\" (UniqueName: \"kubernetes.io/projected/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5-kube-api-access-2zlwl\") pod \"test-curl-6900-af0c54\" (UID: \"7d63d7ad-0c11-400d-beea-dd7fcb7a86a5\") " pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:01.047254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.047221    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:01.413191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.413098    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6900-af0c54"]
Nov 02 15:22:01.736781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.736684    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:01.864222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.864185    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6900-af0c54" event={"ID":"7d63d7ad-0c11-400d-beea-dd7fcb7a86a5","Type":"ContainerStarted","Data":"ce417d41a61d852e5c4bd1f406f7868b1690bde404f6d149241ddd055191750b"}
Nov 02 15:22:01.864222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.864226    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6900-af0c54" event={"ID":"7d63d7ad-0c11-400d-beea-dd7fcb7a86a5","Type":"ContainerStarted","Data":"bad186f52b1b7844d16fa8fd7122fc23a6d5561ee4b196f425ddf0356156509c"}
Nov 02 15:22:01.914400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.914332    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6900-af0c54" podStartSLOduration=1.914312891 podStartE2EDuration="1.914312891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:01.911071474 +0000 UTC m=+1516.165025101" watchObservedRunningTime="2026-11-02 15:22:01.914312891 +0000 UTC m=+1516.168266520"
Nov 02 15:22:01.955660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.955620    7019 patch_prober.go:29] interesting pod/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:22:01.955848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:01.955689    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 15:22:02.520480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:02.520438    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:02.699673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:02.695074    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:02.956085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:02.956047    7019 patch_prober.go:29] interesting pod/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:22:02.956704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:02.956628    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 15:22:03.445511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.420338    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:03.589685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.588018    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"]
Nov 02 15:22:03.683676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.682749    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:03.686682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.685082    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.751764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.739837    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-21ece2-openshift-default-dockercfg-xwxnk\""
Nov 02 15:22:03.751764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.745798    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6900-af0c54"]
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855010    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-podinfo\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855082    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-token\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855148    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-certs\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855187    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istiod-ca-cert\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855284    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-socket\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855317    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-envoy\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855359    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggfs\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-kube-api-access-4ggfs\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855462    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-credential-socket\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.868671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.855714    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-data\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.960360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.960318    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-socket\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.961049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.961027    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-envoy\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.961219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.961199    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggfs\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-kube-api-access-4ggfs\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.961430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.961414    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-credential-socket\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.961764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.961746    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-data\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.961937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.961922    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-podinfo\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.962092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.962069    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-token\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.962241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.962221    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-certs\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.962357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.962343    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istiod-ca-cert\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.963034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.962972    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-socket\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.963373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.963354    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-credential-socket\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.963696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.963672    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istiod-ca-cert\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.963878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.963861    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-data\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.963980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.960618    7019 patch_prober.go:29] interesting pod/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:22:03.964107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.964085    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 15:22:03.964872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.964852    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-certs\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:03.968623 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.968592    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6900-af0c54" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" containerName="test-curl-6900-af0c54" containerID="cri-o://ce417d41a61d852e5c4bd1f406f7868b1690bde404f6d149241ddd055191750b" gracePeriod=2
Nov 02 15:22:03.999672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:03.988325    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-envoy\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:04.018726 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:04.001759    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-af0c54\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:04.021563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.021522    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggfs\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-kube-api-access-4ggfs\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:04.041907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.026200    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-token\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:04.041907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.028145    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-podinfo\") pod \"e2e-api-b-21ece2-openshift-default-548fc68679-48gcf\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") " pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:04.076674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.042793    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6900-af0c54"]
Nov 02 15:22:04.076674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.042850    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"]
Nov 02 15:22:04.076674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.042894    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"]
Nov 02 15:22:04.076674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.044315    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" containerName="test-curl-6900-af0c54"
Nov 02 15:22:04.076674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.044338    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" containerName="test-curl-6900-af0c54"
Nov 02 15:22:04.076674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.044542    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" containerName="test-curl-6900-af0c54"
Nov 02 15:22:04.098822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.091864    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"]
Nov 02 15:22:04.098822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.092025    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.127961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.127928    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:22:04.128302 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:04.128274    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-af0c54\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:04.128699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.128677    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:22:04.140754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.139343    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:04.170281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.170249    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.170782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.170742    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l29v8\" (UniqueName: \"kubernetes.io/projected/ac02fbba-6baa-4f81-a07a-916269af16db-kube-api-access-l29v8\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.173267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.171114    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.281182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.281083    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:04.285272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.285233    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.285599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.285575    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l29v8\" (UniqueName: \"kubernetes.io/projected/ac02fbba-6baa-4f81-a07a-916269af16db-kube-api-access-l29v8\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.285752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.285735    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.369879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.341830    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.392672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.378053    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.392672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.381368    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l29v8\" (UniqueName: \"kubernetes.io/projected/ac02fbba-6baa-4f81-a07a-916269af16db-kube-api-access-l29v8\") pod \"maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.640896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.625606    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:04.859702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.859221    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"]
Nov 02 15:22:04.869464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.868226    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:04.869464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.868326    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:04.869464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.868368    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:04.956696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.956280    7019 patch_prober.go:29] interesting pod/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 15:22:04.956696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.956337    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 15:22:04.987277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:04.981182    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:05.134673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:05.131666    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" event={"ID":"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b","Type":"ContainerStarted","Data":"34afdd65060199dee4148befda1afd9e51c82673c17e47b1849304ba9c61c4b8"}
Nov 02 15:22:05.413912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:05.413879    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"]
Nov 02 15:22:05.415316 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:22:05.415282    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac02fbba_6baa_4f81_a07a_916269af16db.slice/crio-8a4620d4a2c0c08b186b082b0d946027a1b4c78523609ceeccda35b31c76234d WatchSource:0}: Error finding container 8a4620d4a2c0c08b186b082b0d946027a1b4c78523609ceeccda35b31c76234d: Status 404 returned error can't find the container with id 8a4620d4a2c0c08b186b082b0d946027a1b4c78523609ceeccda35b31c76234d
Nov 02 15:22:05.812427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:05.812401    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:22:05.827941 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:05.827903    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-af0c54\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:05.960476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:05.960440    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:22:05.960911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:05.960893    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:22:05.961549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:05.961530    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:22:06.150156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.150117    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2" event={"ID":"ac02fbba-6baa-4f81-a07a-916269af16db","Type":"ContainerStarted","Data":"8365b283cb66f45ca32e4a8331ff482920fdfde2572a38d8ddcef9197055ec2f"}
Nov 02 15:22:06.152910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.152876    7019 generic.go:356] "Generic (PLEG): container finished" podID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" containerID="ce417d41a61d852e5c4bd1f406f7868b1690bde404f6d149241ddd055191750b" exitCode=137
Nov 02 15:22:06.157053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.157011    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2" event={"ID":"ac02fbba-6baa-4f81-a07a-916269af16db","Type":"ContainerStarted","Data":"8a4620d4a2c0c08b186b082b0d946027a1b4c78523609ceeccda35b31c76234d"}
Nov 02 15:22:06.158138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.158079    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" event={"ID":"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b","Type":"ContainerStarted","Data":"2fc8126cfd2e9eda127011658fac014bb0dbc2df05affd87f389775b2c79eb51"}
Nov 02 15:22:06.209161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.209110    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2" podStartSLOduration=3.209096348 podStartE2EDuration="3.209096348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:06.204478925 +0000 UTC m=+1520.458432550" watchObservedRunningTime="2026-11-02 15:22:06.209096348 +0000 UTC m=+1520.463049972"
Nov 02 15:22:06.260211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.260146    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podStartSLOduration=3.260123867 podStartE2EDuration="3.260123867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:06.256059743 +0000 UTC m=+1520.510013368" watchObservedRunningTime="2026-11-02 15:22:06.260123867 +0000 UTC m=+1520.514077492"
Nov 02 15:22:06.277138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.277110    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:06.284265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.284233    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:06.292287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.292210    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:06.293162 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:06.293134    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-af0c54\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:06.319666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.319610    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5-kube-api-access-2zlwl\" (UniqueName: \"kubernetes.io/projected/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5-kube-api-access-2zlwl\") pod \"7d63d7ad-0c11-400d-beea-dd7fcb7a86a5\" (UID: \"7d63d7ad-0c11-400d-beea-dd7fcb7a86a5\") "
Nov 02 15:22:06.321728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.321700    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5-kube-api-access-2zlwl" pod "7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" (UID: "7d63d7ad-0c11-400d-beea-dd7fcb7a86a5"). InnerVolumeSpecName "kube-api-access-2zlwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:06.358084 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:06.358044    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-af0c54\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:06.424371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.424320    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlwl\" (UniqueName: \"kubernetes.io/projected/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5-kube-api-access-2zlwl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:06.627419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:06.627383    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:07.164992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:07.164949    7019 generic.go:356] "Generic (PLEG): container finished" podID="ac02fbba-6baa-4f81-a07a-916269af16db" containerID="8365b283cb66f45ca32e4a8331ff482920fdfde2572a38d8ddcef9197055ec2f" exitCode=0
Nov 02 15:22:07.165429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:07.165032    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2" event={"ID":"ac02fbba-6baa-4f81-a07a-916269af16db","Type":"ContainerDied","Data":"8365b283cb66f45ca32e4a8331ff482920fdfde2572a38d8ddcef9197055ec2f"}
Nov 02 15:22:07.166797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:07.166776    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:07.166904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:07.166836    7019 scope.go:122] "RemoveContainer" containerID="ce417d41a61d852e5c4bd1f406f7868b1690bde404f6d149241ddd055191750b"
Nov 02 15:22:07.167575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:07.167396    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:07.168492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:07.168471    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:22:07.180702 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:07.180671    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-af0c54\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:07.242963 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:07.242929    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-af0c54\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" pod="opendatahub/test-curl-6900-af0c54"
Nov 02 15:22:07.863305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:07.863272    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d63d7ad-0c11-400d-beea-dd7fcb7a86a5" path="/var/lib/kubelet/pods/7d63d7ad-0c11-400d-beea-dd7fcb7a86a5/volumes"
Nov 02 15:22:08.373059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.373034    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:08.403323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.403287    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:08.455609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.455518    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-xks-ca\") pod \"ac02fbba-6baa-4f81-a07a-916269af16db\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") "
Nov 02 15:22:08.455609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.455574    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-openshift-ca\") pod \"ac02fbba-6baa-4f81-a07a-916269af16db\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") "
Nov 02 15:22:08.457292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.456408    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-openshift-ca" pod "ac02fbba-6baa-4f81-a07a-916269af16db" (UID: "ac02fbba-6baa-4f81-a07a-916269af16db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:08.457292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.456767    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac02fbba-6baa-4f81-a07a-916269af16db-kube-api-access-l29v8\" (UniqueName: \"kubernetes.io/projected/ac02fbba-6baa-4f81-a07a-916269af16db-kube-api-access-l29v8\") pod \"ac02fbba-6baa-4f81-a07a-916269af16db\" (UID: \"ac02fbba-6baa-4f81-a07a-916269af16db\") "
Nov 02 15:22:08.457565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.457543    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:08.458475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.458437    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-xks-ca" pod "ac02fbba-6baa-4f81-a07a-916269af16db" (UID: "ac02fbba-6baa-4f81-a07a-916269af16db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:08.461158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.461124    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac02fbba-6baa-4f81-a07a-916269af16db-kube-api-access-l29v8" pod "ac02fbba-6baa-4f81-a07a-916269af16db" (UID: "ac02fbba-6baa-4f81-a07a-916269af16db"). InnerVolumeSpecName "kube-api-access-l29v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:08.559123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.559089    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac02fbba-6baa-4f81-a07a-916269af16db-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:08.559123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.559119    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l29v8\" (UniqueName: \"kubernetes.io/projected/ac02fbba-6baa-4f81-a07a-916269af16db-kube-api-access-l29v8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:08.749471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.749383    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:08.929777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:08.929742    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:09.209343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:09.209311    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"
Nov 02 15:22:09.209550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:09.209359    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2" event={"ID":"ac02fbba-6baa-4f81-a07a-916269af16db","Type":"ContainerDied","Data":"8a4620d4a2c0c08b186b082b0d946027a1b4c78523609ceeccda35b31c76234d"}
Nov 02 15:22:09.209550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:09.209417    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a4620d4a2c0c08b186b082b0d946027a1b4c78523609ceeccda35b31c76234d"
Nov 02 15:22:10.446436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.446407    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:10.595395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.595357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:10.761720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.751441    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"]
Nov 02 15:22:10.761720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.752607    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ac02fbba-6baa-4f81-a07a-916269af16db" containerName="revoke-keys"
Nov 02 15:22:10.761720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.752631    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac02fbba-6baa-4f81-a07a-916269af16db" containerName="revoke-keys"
Nov 02 15:22:10.761720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.752893    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac02fbba-6baa-4f81-a07a-916269af16db" containerName="revoke-keys"
Nov 02 15:22:10.775672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.775040    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:10.814013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.813974    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:10.814444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.814420    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jtm\" (UniqueName: \"kubernetes.io/projected/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-kube-api-access-k2jtm\") pod \"payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:10.814629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.814536    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"]
Nov 02 15:22:10.815053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.815035    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-21ece2\""
Nov 02 15:22:10.816152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.816104    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-21ece2-dockercfg-tnpbd\""
Nov 02 15:22:10.959736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.957062    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:10.959736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.957229    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jtm\" (UniqueName: \"kubernetes.io/projected/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-kube-api-access-k2jtm\") pod \"payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:10.959736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.959292    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:11.008667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:10.995360    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jtm\" (UniqueName: \"kubernetes.io/projected/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-kube-api-access-k2jtm\") pod \"payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:11.182638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:11.182608    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:11.523661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:11.523617    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"]
Nov 02 15:22:11.524886 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:22:11.524862    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2309c81c_8ca6_4ebf_beef_eff4bd1ad5d9.slice/crio-5ccdd9362250d40d1cfb824c452d6e1045284c160327de92118c5a70935fc1f2 WatchSource:0}: Error finding container 5ccdd9362250d40d1cfb824c452d6e1045284c160327de92118c5a70935fc1f2: Status 404 returned error can't find the container with id 5ccdd9362250d40d1cfb824c452d6e1045284c160327de92118c5a70935fc1f2
Nov 02 15:22:12.243383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:12.243337    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq" event={"ID":"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9","Type":"ContainerStarted","Data":"76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5"}
Nov 02 15:22:12.243597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:12.243424    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:12.243597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:12.243448    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq" event={"ID":"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9","Type":"ContainerStarted","Data":"5ccdd9362250d40d1cfb824c452d6e1045284c160327de92118c5a70935fc1f2"}
Nov 02 15:22:12.287443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:12.287382    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq" podStartSLOduration=2.287365458 podStartE2EDuration="2.287365458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:12.286588519 +0000 UTC m=+1526.540542160" watchObservedRunningTime="2026-11-02 15:22:12.287365458 +0000 UTC m=+1526.541319082"
Nov 02 15:22:12.327612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:12.327581    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:13.041830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:13.041792    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"]
Nov 02 15:22:13.055227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:13.055196    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b253fd2f-xppqp"]
Nov 02 15:22:13.864718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:13.864680    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63ef6d7-4a05-41a8-b19f-13be95f965dd" path="/var/lib/kubelet/pods/e63ef6d7-4a05-41a8-b19f-13be95f965dd/volumes"
Nov 02 15:22:13.952849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:13.952408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:14.242234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:14.242145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:14.384929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:14.384893    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:14.594167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:14.594093    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:22:16.404228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:16.404180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:16.769374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:16.769279    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"]
Nov 02 15:22:16.769864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:16.769835    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy" containerID="cri-o://e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408" gracePeriod=30
Nov 02 15:22:18.228774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:18.228739    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:20.083428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:20.083395    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:20.781323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:20.781269    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"]
Nov 02 15:22:20.781617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:20.781559    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" containerID="cri-o://01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730" gracePeriod=30
Nov 02 15:22:21.018799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:21.018758    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:21.910373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:21.910310    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:22.116085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.116057    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:22:22.227359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227324    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-certs\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227390    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-kube-api-access-6qjc6\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-kube-api-access-6qjc6\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227424    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-socket\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227467    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-data\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227517    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-envoy\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227607    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdd1809-2911-4280-8828-00a2fb1569fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd1809-2911-4280-8828-00a2fb1569fb-istiod-ca-cert\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227704    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-certs" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.227835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227805    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-token\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227936    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-credential-socket\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.227995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227937    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-data" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.227995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.227978    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-podinfo\") pod \"2bdd1809-2911-4280-8828-00a2fb1569fb\" (UID: \"2bdd1809-2911-4280-8828-00a2fb1569fb\") "
Nov 02 15:22:22.228147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.228063    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-socket" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.228282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.228260    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdd1809-2911-4280-8828-00a2fb1569fb-istiod-ca-cert" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:22.229002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.228973    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd1809-2911-4280-8828-00a2fb1569fb-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.229117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.229006    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.229117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.229020    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.229117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.229033    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.229319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.229258    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-credential-socket" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.230365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.230336    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-kube-api-access-6qjc6" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "kube-api-access-6qjc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:22.230975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.230947    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-token" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:22.231079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.230971    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-podinfo" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:22.231250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.231221    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-envoy" pod "2bdd1809-2911-4280-8828-00a2fb1569fb" (UID: "2bdd1809-2911-4280-8828-00a2fb1569fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:22.330466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.330431    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.330466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.330462    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.330466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.330471    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.330708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.330480    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjc6\" (UniqueName: \"kubernetes.io/projected/2bdd1809-2911-4280-8828-00a2fb1569fb-kube-api-access-6qjc6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.330708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.330488    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd1809-2911-4280-8828-00a2fb1569fb-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.397120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.397085    7019 generic.go:356] "Generic (PLEG): container finished" podID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerID="e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408" exitCode=0
Nov 02 15:22:22.397268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.397132    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"
Nov 02 15:22:22.397268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.397150    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" event={"ID":"2bdd1809-2911-4280-8828-00a2fb1569fb","Type":"ContainerDied","Data":"e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408"}
Nov 02 15:22:22.397268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.397193    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv" event={"ID":"2bdd1809-2911-4280-8828-00a2fb1569fb","Type":"ContainerDied","Data":"e6954caa8a7ddb2cb0274095afb0086e8cc6e767818f02430e93517b80e6edb3"}
Nov 02 15:22:22.397268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.397222    7019 scope.go:122] "RemoveContainer" containerID="e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408"
Nov 02 15:22:22.409580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.409559    7019 scope.go:122] "RemoveContainer" containerID="e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408"
Nov 02 15:22:22.409959 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:22.409936    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408\": container with ID starting with e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408 not found: ID does not exist" containerID="e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408"
Nov 02 15:22:22.410038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.409967    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408"} err="failed to get container status \"e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408\": rpc error: code = NotFound desc = could not find container \"e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408\": container with ID starting with e8164c3de3eb8143d118984a77a4dff78a64041aabf545999705d55f22927408 not found: ID does not exist"
Nov 02 15:22:22.456706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.456633    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"]
Nov 02 15:22:22.460414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:22.460387    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-w5qzv"]
Nov 02 15:22:22.708106 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:22.708014    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:22:23.166953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.166918    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"]
Nov 02 15:22:23.167703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.167672    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy"
Nov 02 15:22:23.167703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.167689    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy"
Nov 02 15:22:23.167826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.167816    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" containerName="istio-proxy"
Nov 02 15:22:23.171238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.171220    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.190859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.190833    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"]
Nov 02 15:22:23.216895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.216850    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:22:23.217063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.216925    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:22:23.246085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.246047    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.246283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.246263    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7cjd\" (UniqueName: \"kubernetes.io/projected/ad724f70-ce95-4ae7-b751-74261b3b959d-kube-api-access-n7cjd\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.246362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.246312    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.255310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.255289    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:22:23.348411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.348377    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7cjd\" (UniqueName: \"kubernetes.io/projected/ad724f70-ce95-4ae7-b751-74261b3b959d-kube-api-access-n7cjd\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.348616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.348447    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.348714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.348627    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.349333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.349305    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.351348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.351319    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.373376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.373350    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7cjd\" (UniqueName: \"kubernetes.io/projected/ad724f70-ce95-4ae7-b751-74261b3b959d-kube-api-access-n7cjd\") pod \"maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.485739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.485625    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:23.678829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.678793    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:23.800463 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:22:23.800429    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad724f70_ce95_4ae7_b751_74261b3b959d.slice/crio-35c09f129929ac415d574ea14003aae428cfbade00faf912b5be8d0259db4e7d WatchSource:0}: Error finding container 35c09f129929ac415d574ea14003aae428cfbade00faf912b5be8d0259db4e7d: Status 404 returned error can't find the container with id 35c09f129929ac415d574ea14003aae428cfbade00faf912b5be8d0259db4e7d
Nov 02 15:22:23.802831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.802802    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"]
Nov 02 15:22:23.864616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:23.864586    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdd1809-2911-4280-8828-00a2fb1569fb" path="/var/lib/kubelet/pods/2bdd1809-2911-4280-8828-00a2fb1569fb/volumes"
Nov 02 15:22:24.016901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:24.016862    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:22:24.017058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:24.016923    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:22:24.445058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:24.444964    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z" event={"ID":"ad724f70-ce95-4ae7-b751-74261b3b959d","Type":"ContainerStarted","Data":"b085c93cf4ee6df464b80d3a6620bf71dc6fd2fff737244a51da5e7ab3c42cdd"}
Nov 02 15:22:24.445058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:24.445031    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z" event={"ID":"ad724f70-ce95-4ae7-b751-74261b3b959d","Type":"ContainerStarted","Data":"35c09f129929ac415d574ea14003aae428cfbade00faf912b5be8d0259db4e7d"}
Nov 02 15:22:24.489528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:24.489478    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z" podStartSLOduration=1.489466008 podStartE2EDuration="1.489466008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:24.488057425 +0000 UTC m=+1538.742011050" watchObservedRunningTime="2026-11-02 15:22:24.489466008 +0000 UTC m=+1538.743419633"
Nov 02 15:22:25.453240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:25.453204    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:25.457696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:25.457636    7019 generic.go:356] "Generic (PLEG): container finished" podID="ad724f70-ce95-4ae7-b751-74261b3b959d" containerID="b085c93cf4ee6df464b80d3a6620bf71dc6fd2fff737244a51da5e7ab3c42cdd" exitCode=0
Nov 02 15:22:25.457825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:25.457684    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z" event={"ID":"ad724f70-ce95-4ae7-b751-74261b3b959d","Type":"ContainerDied","Data":"b085c93cf4ee6df464b80d3a6620bf71dc6fd2fff737244a51da5e7ab3c42cdd"}
Nov 02 15:22:26.083925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.083898    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:22:26.196914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.196874    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-kube-api-access-c5mb8\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-kube-api-access-c5mb8\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.196942    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-envoy\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.197026    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-podinfo\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.197167    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-token\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-token\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.197210    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec5ddd8-f388-4762-9fd9-ba1231de0867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5ddd8-f388-4762-9fd9-ba1231de0867-istiod-ca-cert\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.197245    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-socket\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.197314    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-certs\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.197344    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-data\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.197568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.197500    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-credential-socket\") pod \"dec5ddd8-f388-4762-9fd9-ba1231de0867\" (UID: \"dec5ddd8-f388-4762-9fd9-ba1231de0867\") "
Nov 02 15:22:26.198989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.198817    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-credential-socket" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:26.199092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.199032    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-socket" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:26.199622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.199593    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-data" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:26.199740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.199712    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-certs" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:26.200256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.200228    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec5ddd8-f388-4762-9fd9-ba1231de0867-istiod-ca-cert" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:26.201357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.201332    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-token" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:26.202895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.202867    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-kube-api-access-c5mb8" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "kube-api-access-c5mb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:26.203302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.203276    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-envoy" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:26.204595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.204561    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-podinfo" pod "dec5ddd8-f388-4762-9fd9-ba1231de0867" (UID: "dec5ddd8-f388-4762-9fd9-ba1231de0867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300409    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5mb8\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-kube-api-access-c5mb8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300442    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300456    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300471    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300483    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dec5ddd8-f388-4762-9fd9-ba1231de0867-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300494    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300506    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300521    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.301290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.300532    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dec5ddd8-f388-4762-9fd9-ba1231de0867-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.468174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.468139    7019 generic.go:356] "Generic (PLEG): container finished" podID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerID="01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730" exitCode=0
Nov 02 15:22:26.468599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.468205    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"
Nov 02 15:22:26.468599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.468204    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" event={"ID":"dec5ddd8-f388-4762-9fd9-ba1231de0867","Type":"ContainerDied","Data":"01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730"}
Nov 02 15:22:26.468599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.468272    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc" event={"ID":"dec5ddd8-f388-4762-9fd9-ba1231de0867","Type":"ContainerDied","Data":"29fb004f8ee33a9f9dbc58887a8f6a2bd7f9e864799fd556f42b2c3b3381ef0d"}
Nov 02 15:22:26.468599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.468302    7019 scope.go:122] "RemoveContainer" containerID="01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730"
Nov 02 15:22:26.481149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.481124    7019 scope.go:122] "RemoveContainer" containerID="73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60"
Nov 02 15:22:26.493149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.493132    7019 scope.go:122] "RemoveContainer" containerID="01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730"
Nov 02 15:22:26.493408 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:26.493388    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730\": container with ID starting with 01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730 not found: ID does not exist" containerID="01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730"
Nov 02 15:22:26.493475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.493419    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730"} err="failed to get container status \"01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730\": rpc error: code = NotFound desc = could not find container \"01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730\": container with ID starting with 01589f20f3aa529a8c953505b5cea1947cc1e9eac70fedf0e082beda30a6f730 not found: ID does not exist"
Nov 02 15:22:26.493475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.493436    7019 scope.go:122] "RemoveContainer" containerID="73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60"
Nov 02 15:22:26.493707 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:26.493689    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60\": container with ID starting with 73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60 not found: ID does not exist" containerID="73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60"
Nov 02 15:22:26.493770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.493709    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60"} err="failed to get container status \"73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60\": rpc error: code = NotFound desc = could not find container \"73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60\": container with ID starting with 73714736c98a6c55b622e9947eb113ee4ea884eacf2914fc6c0f56f502646c60 not found: ID does not exist"
Nov 02 15:22:26.610878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.610850    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"]
Nov 02 15:22:26.636786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.636757    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-2tnfc"]
Nov 02 15:22:26.642019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.641999    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:26.709149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.709113    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"]
Nov 02 15:22:26.709385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.709352    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy" containerID="cri-o://9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea" gracePeriod=30
Nov 02 15:22:26.710151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.710109    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad724f70-ce95-4ae7-b751-74261b3b959d-kube-api-access-n7cjd\" (UniqueName: \"kubernetes.io/projected/ad724f70-ce95-4ae7-b751-74261b3b959d-kube-api-access-n7cjd\") pod \"ad724f70-ce95-4ae7-b751-74261b3b959d\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") "
Nov 02 15:22:26.710230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.710163    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-openshift-ca\") pod \"ad724f70-ce95-4ae7-b751-74261b3b959d\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") "
Nov 02 15:22:26.710230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.710209    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-xks-ca\") pod \"ad724f70-ce95-4ae7-b751-74261b3b959d\" (UID: \"ad724f70-ce95-4ae7-b751-74261b3b959d\") "
Nov 02 15:22:26.711896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.711867    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-openshift-ca" pod "ad724f70-ce95-4ae7-b751-74261b3b959d" (UID: "ad724f70-ce95-4ae7-b751-74261b3b959d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:26.716494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.714391    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-xks-ca" pod "ad724f70-ce95-4ae7-b751-74261b3b959d" (UID: "ad724f70-ce95-4ae7-b751-74261b3b959d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:26.716494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.714587    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"]
Nov 02 15:22:26.716494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.714807    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy" containerID="cri-o://ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e" gracePeriod=30
Nov 02 15:22:26.716907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.716878    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad724f70-ce95-4ae7-b751-74261b3b959d-kube-api-access-n7cjd" pod "ad724f70-ce95-4ae7-b751-74261b3b959d" (UID: "ad724f70-ce95-4ae7-b751-74261b3b959d"). InnerVolumeSpecName "kube-api-access-n7cjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:26.813034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.813006    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7cjd\" (UniqueName: \"kubernetes.io/projected/ad724f70-ce95-4ae7-b751-74261b3b959d-kube-api-access-n7cjd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.813034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.813032    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:26.813251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:26.813042    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad724f70-ce95-4ae7-b751-74261b3b959d-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:27.300865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:27.300822    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:27.480408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:27.480244    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z" event={"ID":"ad724f70-ce95-4ae7-b751-74261b3b959d","Type":"ContainerDied","Data":"35c09f129929ac415d574ea14003aae428cfbade00faf912b5be8d0259db4e7d"}
Nov 02 15:22:27.480408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:27.480280    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"
Nov 02 15:22:27.480408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:27.480318    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35c09f129929ac415d574ea14003aae428cfbade00faf912b5be8d0259db4e7d"
Nov 02 15:22:27.863374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:27.863339    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" path="/var/lib/kubelet/pods/dec5ddd8-f388-4762-9fd9-ba1231de0867/volumes"
Nov 02 15:22:29.031679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:29.020925    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:30.714557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:30.714426    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:32.061374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.061352    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:22:32.081904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.081878    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:22:32.217393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217354    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-envoy\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.217565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217425    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc15e21c-0124-4eff-95e5-234166fb9afc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc15e21c-0124-4eff-95e5-234166fb9afc-istiod-ca-cert\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.217565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217489    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-envoy\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.217565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217517    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-data\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.217731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217560    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-kube-api-access-fw245\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-kube-api-access-fw245\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.217793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217755    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-podinfo\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.217883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217862    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-podinfo\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.217883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217864    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-data" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.218004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217889    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc15e21c-0124-4eff-95e5-234166fb9afc-istiod-ca-cert" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:32.218004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217925    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-token\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-token\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.218004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.217981    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-kube-api-access-29sc2\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-kube-api-access-29sc2\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.218222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218130    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istiod-ca-cert\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.218222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218216    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-token\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-token\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.218338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218246    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-socket\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.218338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218306    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-certs\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.218442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218407    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-certs\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.218496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218470    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-data\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.218550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218501    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-credential-socket\") pod \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\" (UID: \"f29800bd-eb81-4d8b-8561-51eb0c07cc2b\") "
Nov 02 15:22:32.218603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218589    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-credential-socket\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.218706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.218634    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-socket\") pod \"fc15e21c-0124-4eff-95e5-234166fb9afc\" (UID: \"fc15e21c-0124-4eff-95e5-234166fb9afc\") "
Nov 02 15:22:32.219601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.219571    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc15e21c-0124-4eff-95e5-234166fb9afc-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.219736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.219605    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.219920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.219875    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-socket" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.220287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.220256    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istiod-ca-cert" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:32.220495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.220474    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-kube-api-access-fw245" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "kube-api-access-fw245". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.220641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.220619    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-envoy" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.220786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.220716    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-certs" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.220905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.220884    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-envoy" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.221003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.220984    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-podinfo" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:32.221164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.221145    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-credential-socket" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.221240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.221178    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-credential-socket" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.221419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.221258    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-certs" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.221419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.221349    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-data" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.221419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.221357    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-socket" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:22:32.221696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.221508    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-podinfo" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:22:32.221892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.221867    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-kube-api-access-29sc2" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "kube-api-access-29sc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.222110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.222081    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-token" pod "fc15e21c-0124-4eff-95e5-234166fb9afc" (UID: "fc15e21c-0124-4eff-95e5-234166fb9afc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.222444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.222423    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-token" pod "f29800bd-eb81-4d8b-8561-51eb0c07cc2b" (UID: "f29800bd-eb81-4d8b-8561-51eb0c07cc2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321108    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321136    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321146    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29sc2\" (UniqueName: \"kubernetes.io/projected/fc15e21c-0124-4eff-95e5-234166fb9afc-kube-api-access-29sc2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321156    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321165    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321172    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321180    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321188    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321196    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321203    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321211    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321219    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321227    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc15e21c-0124-4eff-95e5-234166fb9afc-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321238    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321246    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw245\" (UniqueName: \"kubernetes.io/projected/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-kube-api-access-fw245\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.321549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.321253    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f29800bd-eb81-4d8b-8561-51eb0c07cc2b-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:32.374758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.374715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:32.556859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.556822    7019 generic.go:356] "Generic (PLEG): container finished" podID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerID="ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e" exitCode=0
Nov 02 15:22:32.557044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.556947    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" event={"ID":"fc15e21c-0124-4eff-95e5-234166fb9afc","Type":"ContainerDied","Data":"ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e"}
Nov 02 15:22:32.557044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.556997    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6" event={"ID":"fc15e21c-0124-4eff-95e5-234166fb9afc","Type":"ContainerDied","Data":"de27c4e6ca4ca51f831842bc45927f22e3d3371b8d62309b663a4e15ab20e014"}
Nov 02 15:22:32.557044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.557026    7019 scope.go:122] "RemoveContainer" containerID="ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e"
Nov 02 15:22:32.557309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.557258    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"
Nov 02 15:22:32.560608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.560579    7019 generic.go:356] "Generic (PLEG): container finished" podID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerID="9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea" exitCode=0
Nov 02 15:22:32.560893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.560709    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" event={"ID":"f29800bd-eb81-4d8b-8561-51eb0c07cc2b","Type":"ContainerDied","Data":"9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea"}
Nov 02 15:22:32.560893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.560750    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct" event={"ID":"f29800bd-eb81-4d8b-8561-51eb0c07cc2b","Type":"ContainerDied","Data":"ba312c305f3f02e89a2c187907b4e0fc90d2c575034b22439b16b2d0d405ec84"}
Nov 02 15:22:32.560893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.560818    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"
Nov 02 15:22:32.571663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.571591    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:32.592743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.592714    7019 scope.go:122] "RemoveContainer" containerID="ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e"
Nov 02 15:22:32.593549 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:32.593169    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e\": container with ID starting with ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e not found: ID does not exist" containerID="ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e"
Nov 02 15:22:32.593549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.593212    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e"} err="failed to get container status \"ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e\": rpc error: code = NotFound desc = could not find container \"ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e\": container with ID starting with ac9bb1924a7a8531a3d9852b22923cb0f0678ae58ed6eddf858d868499444c8e not found: ID does not exist"
Nov 02 15:22:32.593549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.593237    7019 scope.go:122] "RemoveContainer" containerID="9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea"
Nov 02 15:22:32.612703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.610404    7019 scope.go:122] "RemoveContainer" containerID="9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea"
Nov 02 15:22:32.612703 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:32.610816    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea\": container with ID starting with 9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea not found: ID does not exist" containerID="9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea"
Nov 02 15:22:32.614042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.610855    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea"} err="failed to get container status \"9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea\": rpc error: code = NotFound desc = could not find container \"9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea\": container with ID starting with 9a267f25de8e1c042e31c02bf8e9c2df181875df0ac1e081435e35a565a88eea not found: ID does not exist"
Nov 02 15:22:32.624102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.624068    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"]
Nov 02 15:22:32.630179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.630152    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-72rl6"]
Nov 02 15:22:32.686520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.686417    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"]
Nov 02 15:22:32.690447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.690421    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-vx5ct"]
Nov 02 15:22:32.733584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:32.733550    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:33.414958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:33.414920    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"]
Nov 02 15:22:33.552407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:33.551700    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"]
Nov 02 15:22:33.641313 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:33.641278    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:33.793253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:33.793167    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:33.892705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:33.892664    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" path="/var/lib/kubelet/pods/f29800bd-eb81-4d8b-8561-51eb0c07cc2b/volumes"
Nov 02 15:22:33.894002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:33.893310    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" path="/var/lib/kubelet/pods/fc15e21c-0124-4eff-95e5-234166fb9afc/volumes"
Nov 02 15:22:34.255669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:34.255618    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:36.106806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:36.106769    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:37.840469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:37.840435    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:38.023674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.018472    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:38.415746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.415705    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2" podUID="2da826e5-1db1-4875-b9ce-5354e2e896be" containerName="payload-processing" containerID="cri-o://6641dd91ccb8ec996b98039642a325795d458e4cfbc3412b390640a466de1195" gracePeriod=25
Nov 02 15:22:38.553038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.552981    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br" podUID="31488e9e-b222-49bb-898b-80aeac7e0481" containerName="payload-pre-processing" containerID="cri-o://e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0" gracePeriod=25
Nov 02 15:22:38.679572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.675119    7019 generic.go:356] "Generic (PLEG): container finished" podID="2da826e5-1db1-4875-b9ce-5354e2e896be" containerID="6641dd91ccb8ec996b98039642a325795d458e4cfbc3412b390640a466de1195" exitCode=0
Nov 02 15:22:38.679572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.675347    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2" event={"ID":"2da826e5-1db1-4875-b9ce-5354e2e896be","Type":"ContainerDied","Data":"6641dd91ccb8ec996b98039642a325795d458e4cfbc3412b390640a466de1195"}
Nov 02 15:22:38.815791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.814735    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:22:38.898685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.888522    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da826e5-1db1-4875-b9ce-5354e2e896be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da826e5-1db1-4875-b9ce-5354e2e896be-plugins-config-volume\") pod \"2da826e5-1db1-4875-b9ce-5354e2e896be\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") "
Nov 02 15:22:38.898685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.888761    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da826e5-1db1-4875-b9ce-5354e2e896be-kube-api-access-hvr4x\" (UniqueName: \"kubernetes.io/projected/2da826e5-1db1-4875-b9ce-5354e2e896be-kube-api-access-hvr4x\") pod \"2da826e5-1db1-4875-b9ce-5354e2e896be\" (UID: \"2da826e5-1db1-4875-b9ce-5354e2e896be\") "
Nov 02 15:22:38.946126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:38.940069    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da826e5-1db1-4875-b9ce-5354e2e896be-plugins-config-volume" pod "2da826e5-1db1-4875-b9ce-5354e2e896be" (UID: "2da826e5-1db1-4875-b9ce-5354e2e896be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:39.060667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.057701    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da826e5-1db1-4875-b9ce-5354e2e896be-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:39.075740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.071522    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da826e5-1db1-4875-b9ce-5354e2e896be-kube-api-access-hvr4x" pod "2da826e5-1db1-4875-b9ce-5354e2e896be" (UID: "2da826e5-1db1-4875-b9ce-5354e2e896be"). InnerVolumeSpecName "kube-api-access-hvr4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:39.105890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.105862    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:22:39.159614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.159578    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvr4x\" (UniqueName: \"kubernetes.io/projected/2da826e5-1db1-4875-b9ce-5354e2e896be-kube-api-access-hvr4x\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:39.262807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.262705    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31488e9e-b222-49bb-898b-80aeac7e0481-kube-api-access-bccms\" (UniqueName: \"kubernetes.io/projected/31488e9e-b222-49bb-898b-80aeac7e0481-kube-api-access-bccms\") pod \"31488e9e-b222-49bb-898b-80aeac7e0481\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") "
Nov 02 15:22:39.262807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.262768    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31488e9e-b222-49bb-898b-80aeac7e0481-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31488e9e-b222-49bb-898b-80aeac7e0481-plugins-config-volume\") pod \"31488e9e-b222-49bb-898b-80aeac7e0481\" (UID: \"31488e9e-b222-49bb-898b-80aeac7e0481\") "
Nov 02 15:22:39.263745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.263711    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31488e9e-b222-49bb-898b-80aeac7e0481-plugins-config-volume" pod "31488e9e-b222-49bb-898b-80aeac7e0481" (UID: "31488e9e-b222-49bb-898b-80aeac7e0481"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:39.265067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.265040    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31488e9e-b222-49bb-898b-80aeac7e0481-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:39.267897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.267862    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31488e9e-b222-49bb-898b-80aeac7e0481-kube-api-access-bccms" pod "31488e9e-b222-49bb-898b-80aeac7e0481" (UID: "31488e9e-b222-49bb-898b-80aeac7e0481"). InnerVolumeSpecName "kube-api-access-bccms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:39.308554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.308523    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"]
Nov 02 15:22:39.308808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.308763    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6" podUID="ba41140f-790b-4fad-b28b-6d19ce0a9c94" containerName="maas-api" containerID="cri-o://c8f58fcd1ece4d066ac97ccf2b25add306f70a6aeee9fe0b6bd0b60e02c14a18" gracePeriod=30
Nov 02 15:22:39.368671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.367207    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bccms\" (UniqueName: \"kubernetes.io/projected/31488e9e-b222-49bb-898b-80aeac7e0481-kube-api-access-bccms\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:39.435512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:39.435479    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:40.054731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.016266    7019 generic.go:356] "Generic (PLEG): container finished" podID="31488e9e-b222-49bb-898b-80aeac7e0481" containerID="e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0" exitCode=0
Nov 02 15:22:40.054731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.016460    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"
Nov 02 15:22:40.054731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.019659    7019 generic.go:356] "Generic (PLEG): container finished" podID="ba41140f-790b-4fad-b28b-6d19ce0a9c94" containerID="c8f58fcd1ece4d066ac97ccf2b25add306f70a6aeee9fe0b6bd0b60e02c14a18" exitCode=0
Nov 02 15:22:40.054731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.025245    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"
Nov 02 15:22:40.087679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.084376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:40.146464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.144025    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br" event={"ID":"31488e9e-b222-49bb-898b-80aeac7e0481","Type":"ContainerDied","Data":"e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0"}
Nov 02 15:22:40.146464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.144136    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br" event={"ID":"31488e9e-b222-49bb-898b-80aeac7e0481","Type":"ContainerDied","Data":"3cf38f17e261db56af9eafe749d248eb7fbdb94b6bf78b6254b38e4f106b1827"}
Nov 02 15:22:40.146464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.144161    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6" event={"ID":"ba41140f-790b-4fad-b28b-6d19ce0a9c94","Type":"ContainerDied","Data":"c8f58fcd1ece4d066ac97ccf2b25add306f70a6aeee9fe0b6bd0b60e02c14a18"}
Nov 02 15:22:40.146464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.144198    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2" event={"ID":"2da826e5-1db1-4875-b9ce-5354e2e896be","Type":"ContainerDied","Data":"5cb79d71c515d4c68b3f435e69e2666366941fb9c3c24ae3369562999096d9f0"}
Nov 02 15:22:40.146464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.144236    7019 scope.go:122] "RemoveContainer" containerID="e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0"
Nov 02 15:22:40.246752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.246502    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:40.251965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.250735    7019 scope.go:122] "RemoveContainer" containerID="e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0"
Nov 02 15:22:40.251965 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:40.251070    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0\": container with ID starting with e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0 not found: ID does not exist" containerID="e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0"
Nov 02 15:22:40.251965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.251102    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0"} err="failed to get container status \"e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0\": rpc error: code = NotFound desc = could not find container \"e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0\": container with ID starting with e836d4dac2da19bf30b58b008a9b2aa2f1ed8752f40728d07a84feff6739baf0 not found: ID does not exist"
Nov 02 15:22:40.251965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.251125    7019 scope.go:122] "RemoveContainer" containerID="6641dd91ccb8ec996b98039642a325795d458e4cfbc3412b390640a466de1195"
Nov 02 15:22:40.256604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.256372    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"]
Nov 02 15:22:40.262761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.261916    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3d7df16e-7j6k2"]
Nov 02 15:22:40.323717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.321774    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"]
Nov 02 15:22:40.333449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.333417    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:22:40.337679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.337496    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55e4c3-c5bc59bb5-cwqd2"]
Nov 02 15:22:40.349731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.349701    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"]
Nov 02 15:22:40.350177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.350153    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55e4c3-55f6c95567-s52br"]
Nov 02 15:22:40.388431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.388392    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:40.519631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.519597    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-metrics-certs\") pod \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") "
Nov 02 15:22:40.519851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.519703    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba41140f-790b-4fad-b28b-6d19ce0a9c94-kube-api-access-qnp6g\" (UniqueName: \"kubernetes.io/projected/ba41140f-790b-4fad-b28b-6d19ce0a9c94-kube-api-access-qnp6g\") pod \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") "
Nov 02 15:22:40.519851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.519833    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-maas-api-tls\") pod \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") "
Nov 02 15:22:40.520019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.520001    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba41140f-790b-4fad-b28b-6d19ce0a9c94-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba41140f-790b-4fad-b28b-6d19ce0a9c94-openshift-service-ca\") pod \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\" (UID: \"ba41140f-790b-4fad-b28b-6d19ce0a9c94\") "
Nov 02 15:22:40.520414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.520384    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba41140f-790b-4fad-b28b-6d19ce0a9c94-openshift-service-ca" pod "ba41140f-790b-4fad-b28b-6d19ce0a9c94" (UID: "ba41140f-790b-4fad-b28b-6d19ce0a9c94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:40.520694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.520630    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba41140f-790b-4fad-b28b-6d19ce0a9c94-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:40.521966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.521941    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-metrics-certs" pod "ba41140f-790b-4fad-b28b-6d19ce0a9c94" (UID: "ba41140f-790b-4fad-b28b-6d19ce0a9c94"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:40.522056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.522010    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba41140f-790b-4fad-b28b-6d19ce0a9c94-kube-api-access-qnp6g" pod "ba41140f-790b-4fad-b28b-6d19ce0a9c94" (UID: "ba41140f-790b-4fad-b28b-6d19ce0a9c94"). InnerVolumeSpecName "kube-api-access-qnp6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:40.522056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.522020    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-maas-api-tls" pod "ba41140f-790b-4fad-b28b-6d19ce0a9c94" (UID: "ba41140f-790b-4fad-b28b-6d19ce0a9c94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:40.621630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.621598    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:40.621630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.621628    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba41140f-790b-4fad-b28b-6d19ce0a9c94-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:40.621838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.621637    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnp6g\" (UniqueName: \"kubernetes.io/projected/ba41140f-790b-4fad-b28b-6d19ce0a9c94-kube-api-access-qnp6g\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:22:40.785142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:40.782467    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:41.035609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.035524    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"
Nov 02 15:22:41.035609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.035567    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6" event={"ID":"ba41140f-790b-4fad-b28b-6d19ce0a9c94","Type":"ContainerDied","Data":"16a779c47ef6a2148327b0b041570813ed67659a81132844a9569044d922a434"}
Nov 02 15:22:41.035864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.035631    7019 scope.go:122] "RemoveContainer" containerID="c8f58fcd1ece4d066ac97ccf2b25add306f70a6aeee9fe0b6bd0b60e02c14a18"
Nov 02 15:22:41.115694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.115657    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"]
Nov 02 15:22:41.145398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.145370    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3d7df16e-b9fb75786-6tgd6"]
Nov 02 15:22:41.865143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.865107    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da826e5-1db1-4875-b9ce-5354e2e896be" path="/var/lib/kubelet/pods/2da826e5-1db1-4875-b9ce-5354e2e896be/volumes"
Nov 02 15:22:41.865706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.865690    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31488e9e-b222-49bb-898b-80aeac7e0481" path="/var/lib/kubelet/pods/31488e9e-b222-49bb-898b-80aeac7e0481/volumes"
Nov 02 15:22:41.866264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.866247    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac02fbba-6baa-4f81-a07a-916269af16db" path="/var/lib/kubelet/pods/ac02fbba-6baa-4f81-a07a-916269af16db/volumes"
Nov 02 15:22:41.867256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:41.867235    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba41140f-790b-4fad-b28b-6d19ce0a9c94" path="/var/lib/kubelet/pods/ba41140f-790b-4fad-b28b-6d19ce0a9c94/volumes"
Nov 02 15:22:42.285964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:42.285876    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:44.162278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:44.156864    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:45.809820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:45.807578    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:46.043729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:46.042064    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:46.955762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:46.955727    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:47.505674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:47.500830    7019 scope.go:122] "RemoveContainer" containerID="f2e52fc105712d1b946e81f94ac1906adefd7272c34e8be5737507c6f95092ee"
Nov 02 15:22:47.510978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:47.510940    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:49.193407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:49.193364    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:51.001450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:51.001416    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:52.506874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.506842    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"]
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507586    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2da826e5-1db1-4875-b9ce-5354e2e896be" containerName="payload-processing"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507601    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da826e5-1db1-4875-b9ce-5354e2e896be" containerName="payload-processing"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507628    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507635    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507659    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ba41140f-790b-4fad-b28b-6d19ce0a9c94" containerName="maas-api"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507664    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba41140f-790b-4fad-b28b-6d19ce0a9c94" containerName="maas-api"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507673    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507677    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507686    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507692    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507702    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ad724f70-ce95-4ae7-b751-74261b3b959d" containerName="revoke-keys"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507706    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad724f70-ce95-4ae7-b751-74261b3b959d" containerName="revoke-keys"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507729    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507734    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507748    7019 cpu_manager.go:414] "containerMap: removing container" podUID="31488e9e-b222-49bb-898b-80aeac7e0481" containerName="payload-pre-processing"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507753    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31488e9e-b222-49bb-898b-80aeac7e0481" containerName="payload-pre-processing"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507855    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="31488e9e-b222-49bb-898b-80aeac7e0481" containerName="payload-pre-processing"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507864    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da826e5-1db1-4875-b9ce-5354e2e896be" containerName="payload-processing"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507875    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc15e21c-0124-4eff-95e5-234166fb9afc" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507881    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba41140f-790b-4fad-b28b-6d19ce0a9c94" containerName="maas-api"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507887    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507893    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec5ddd8-f388-4762-9fd9-ba1231de0867" containerName="istio-proxy"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507899    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad724f70-ce95-4ae7-b751-74261b3b959d" containerName="revoke-keys"
Nov 02 15:22:52.508244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.507908    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29800bd-eb81-4d8b-8561-51eb0c07cc2b" containerName="istio-proxy"
Nov 02 15:22:52.510723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.510707    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.534127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.534098    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"]
Nov 02 15:22:52.541178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.541152    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-21ece2-serving-cert\""
Nov 02 15:22:52.635638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.635603    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:52.677299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.677267    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-metrics-certs\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.677468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.677344    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-maas-api-tls\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.677468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.677455    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vfj\" (UniqueName: \"kubernetes.io/projected/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-kube-api-access-c7vfj\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.677557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.677517    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-openshift-service-ca\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.705641 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:22:52.705571    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:22:52.798729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.795908    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vfj\" (UniqueName: \"kubernetes.io/projected/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-kube-api-access-c7vfj\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.798729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.796044    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-openshift-service-ca\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.798729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.796422    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-metrics-certs\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.799030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.798920    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:52.804668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.800807    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-openshift-service-ca\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.804668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.803734    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-maas-api-tls\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.824798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.811561    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-metrics-certs\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.824798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.818226    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-maas-api-tls\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:52.830673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:52.829107    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vfj\" (UniqueName: \"kubernetes.io/projected/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-kube-api-access-c7vfj\") pod \"maas-api-e2e-api-b-21ece2-6954485c6-prhg5\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:53.124845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.124807    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:53.319434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.319371    7019 generic.go:356] "Generic (PLEG): container finished" podID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerID="775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf" exitCode=0
Nov 02 15:22:53.320041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.319997    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" event={"ID":"377eabb1-c526-4a6b-9d85-05d6004a0cf9","Type":"ContainerDied","Data":"775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf"}
Nov 02 15:22:53.320782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.320760    7019 scope.go:122] "RemoveContainer" containerID="775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf"
Nov 02 15:22:53.323697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.323625    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:53.323905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.323884    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:53.324040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.324023    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:53.449682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:53.449066    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"]
Nov 02 15:22:54.330724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:54.330606    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5" event={"ID":"0e950221-5e22-4d5f-86b3-ed28cf3e1e73","Type":"ContainerStarted","Data":"8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee"}
Nov 02 15:22:54.331187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:54.330814    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:22:54.331187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:54.330837    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5" event={"ID":"0e950221-5e22-4d5f-86b3-ed28cf3e1e73","Type":"ContainerStarted","Data":"bbe9e0086be53634e45bab82c37b3bec74c41c6eb2be64ef34909a1e2762a1fc"}
Nov 02 15:22:54.333789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:54.333758    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" event={"ID":"377eabb1-c526-4a6b-9d85-05d6004a0cf9","Type":"ContainerStarted","Data":"b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0"}
Nov 02 15:22:54.386685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:54.377698    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:54.407687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:54.399949    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5" podStartSLOduration=1.951765774 podStartE2EDuration="2.3999263s" totalImagesPullingTime="448.160526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:54.378110401 +0000 UTC m=+1568.632064026" watchObservedRunningTime="2026-11-02 15:22:54.3999263 +0000 UTC m=+1568.653879925"
Nov 02 15:22:54.545219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:54.545185    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:55.046965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.046904    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:55.188675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.187817    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:22:55.192667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.190761    7019 patch_prober.go:29] interesting pod/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:22:55.192667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.190816    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 15:22:55.521141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.521089    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"]
Nov 02 15:22:55.526957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.526934    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.551576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.551537    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2abeaa-openshift-default-dockercfg-smrsn\""
Nov 02 15:22:55.554362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554328    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554392    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554492    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554543    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554667    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554838    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvv9\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-kube-api-access-jdvv9\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554874    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554910    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.554974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.554961    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:55.590402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:55.590367    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"]
Nov 02 15:22:56.120670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:56.120670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120397    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.120670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120458    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.120670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120524    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.121043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120797    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvv9\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-kube-api-access-jdvv9\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.121043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120833    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.121043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120871    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.121043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.120898    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.121250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.121052    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.121250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.121104    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.128951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.121818    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-data\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.128951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.127139    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.129185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.129095    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-credential-socket\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.129724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.129697    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istiod-ca-cert\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.129971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.129952    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-certs\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.136406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.136232    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-envoy\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.140235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.140208    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-podinfo\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.151214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.151185    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-token\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.154283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.154258    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvv9\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-kube-api-access-jdvv9\") pod \"e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") " pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.180970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.180937    7019 patch_prober.go:29] interesting pod/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:22:56.181172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.181146    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 15:22:56.397456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.397358    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:56.447729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.447690    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:56.553672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.550901    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:56.908620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.902890    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:56.908900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.904446    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"]
Nov 02 15:22:56.908900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.908793    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:56.908900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:56.908857    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:22:57.192729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:57.188210    7019 patch_prober.go:29] interesting pod/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:22:57.192729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:57.188272    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 15:22:57.366132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:57.366094    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" event={"ID":"fa84d2fd-0959-4ef0-baf5-d59c36f24337","Type":"ContainerStarted","Data":"1a71d7181042133f56566a3eebc3b3ba503559008119cafb0993a645a0cad232"}
Nov 02 15:22:57.372838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:57.372806    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" event={"ID":"fa84d2fd-0959-4ef0-baf5-d59c36f24337","Type":"ContainerStarted","Data":"339f07281a368f967621748f8c4ee124d43b87339c33490e9bcef355d4c9393f"}
Nov 02 15:22:57.430311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:57.430255    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podStartSLOduration=2.430241328 podStartE2EDuration="2.430241328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:57.42340356 +0000 UTC m=+1571.677357189" watchObservedRunningTime="2026-11-02 15:22:57.430241328 +0000 UTC m=+1571.684194952"
Nov 02 15:22:57.447948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:57.447861    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:58.177672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:58.177159    7019 patch_prober.go:29] interesting pod/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:22:58.177672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:58.177236    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 15:22:58.217333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:58.217294    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:58.453841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:58.453759    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:59.177680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:59.177010    7019 patch_prober.go:29] interesting pod/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body=
Nov 02 15:22:59.177680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:59.177072    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused"
Nov 02 15:22:59.412206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:59.412180    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:59.413329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:59.413307    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:22:59.785737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:59.785616    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:22:59.939897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:22:59.939859    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:00.086805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:00.086720    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:00.357620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:00.357536    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:23:00.358826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:00.358801    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:23:00.359517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:00.359494    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:23:00.360310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:00.360290    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:23:00.696953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:00.696915    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:01.642113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:01.642058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:01.818340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:01.818300    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:02.021834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:02.021743    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:03.994568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:03.994535    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:05.711414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:05.711374    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:07.483308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:07.483254    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:09.010502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:09.010460    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:09.168309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:09.168273    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:09.336261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:09.336171    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:09.500261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:09.500229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:10.997835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:10.997780    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:12.629764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:12.629721    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:13.922302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:13.922269    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:23:14.613278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:14.613243    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:16.588018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:16.587969    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:18.287245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:18.287200    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:19.986672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:19.980999    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:21.935347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:21.931337    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:22.901283 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:23:22.901240    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:23:23.224625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:23.224522    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:24.012991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:24.012934    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:24.248371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:24.248330    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:25.025171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:25.025134    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:25.458896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:25.458858    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:25.943125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:25.943072    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"]
Nov 02 15:23:25.975978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:25.964191    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089178    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-credential-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089237    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-data\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089283    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-envoy\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089333    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9pt\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-kube-api-access-7f9pt\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089377    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089442    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-token\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089517    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istiod-ca-cert\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089706    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-certs\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.094684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.089737    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-podinfo\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.168685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.131370    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3daba5-openshift-default-dockercfg-gcznk\""
Nov 02 15:23:26.189934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.189714    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.198805    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-credential-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.198863    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-data\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.198906    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-envoy\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.198949    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9pt\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-kube-api-access-7f9pt\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.198987    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.199050    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-token\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.199124    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istiod-ca-cert\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.199278    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-certs\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.199305    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-podinfo\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.213815    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-credential-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.214089    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-data\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.214817    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-socket\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.215785    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istiod-ca-cert\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.215901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.215916    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-certs\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.221988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.221687    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"]
Nov 02 15:23:26.288614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.288543    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-envoy\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.293363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.293281    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-podinfo\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.309974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.309919    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-token\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.323243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.323169    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9pt\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-kube-api-access-7f9pt\") pod \"e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") " pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.343717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.341867    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:26.461735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.461697    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:26.516065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.515992    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:26.812435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.812405    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"]
Nov 02 15:23:26.820172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.820116    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:23:26.820281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.820215    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:23:26.820281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.820260    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:23:26.835632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:26.835596    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" event={"ID":"92d092a7-8dbc-4d1f-a67f-ad9ca8645796","Type":"ContainerStarted","Data":"7c61733d73eea85fa855a9c49afe9d53ed2efe85643a2b17ea2c34567e9c1070"}
Nov 02 15:23:27.847346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.847244    7019 generic.go:356] "Generic (PLEG): container finished" podID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerID="2fc8126cfd2e9eda127011658fac014bb0dbc2df05affd87f389775b2c79eb51" exitCode=0
Nov 02 15:23:27.847346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.847311    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" event={"ID":"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b","Type":"ContainerDied","Data":"2fc8126cfd2e9eda127011658fac014bb0dbc2df05affd87f389775b2c79eb51"}
Nov 02 15:23:27.847941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.847829    7019 scope.go:122] "RemoveContainer" containerID="2fc8126cfd2e9eda127011658fac014bb0dbc2df05affd87f389775b2c79eb51"
Nov 02 15:23:27.849430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.849399    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" event={"ID":"92d092a7-8dbc-4d1f-a67f-ad9ca8645796","Type":"ContainerStarted","Data":"bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984"}
Nov 02 15:23:27.849734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.849707    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:23:27.849812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.849783    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:23:27.849849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.849826    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:23:27.978979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:27.978012    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podStartSLOduration=2.97799157 podStartE2EDuration="2.97799157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:27.972295311 +0000 UTC m=+1602.226248935" watchObservedRunningTime="2026-11-02 15:23:27.97799157 +0000 UTC m=+1602.231945197"
Nov 02 15:23:28.011831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:28.011733    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:28.204410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:28.204352    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:28.462303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:28.462221    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:28.469091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:28.469065    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:28.901671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:28.901078    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" event={"ID":"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b","Type":"ContainerStarted","Data":"1b8d7225bc4d471bf4c7f2f8996817a94a007396fea004250806bddfee60f512"}
Nov 02 15:23:28.901671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:28.901206    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:28.917067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:28.917014    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:23:29.292728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.291748    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:23:29.296161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.296083    7019 patch_prober.go:29] interesting pod/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 15:23:29.296326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.296162    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 15:23:29.461195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.461161    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"]
Nov 02 15:23:29.470670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.468923    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.490936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.488767    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"]
Nov 02 15:23:29.490936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.489223    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:23:29.490936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.490257    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:29.613670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.612880    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.613670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.612938    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvb5\" (UniqueName: \"kubernetes.io/projected/14d47b64-646f-4d92-9c08-1b94841aaba4-kube-api-access-9wvb5\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.613670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.613131    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.714954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.714916    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.715201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.715185    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.715271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.715217    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvb5\" (UniqueName: \"kubernetes.io/projected/14d47b64-646f-4d92-9c08-1b94841aaba4-kube-api-access-9wvb5\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.716069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.716036    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.717717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.717690    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.751606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.751525    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvb5\" (UniqueName: \"kubernetes.io/projected/14d47b64-646f-4d92-9c08-1b94841aaba4-kube-api-access-9wvb5\") pod \"maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.825771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.825736    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:29.901379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:29.901295    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:30.223828 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:23:30.223795    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d47b64_646f_4d92_9c08_1b94841aaba4.slice/crio-bed27599fb721e2e6ff4c9d3a2c409d5288186503aa29b9cd938372e4ba8179f WatchSource:0}: Error finding container bed27599fb721e2e6ff4c9d3a2c409d5288186503aa29b9cd938372e4ba8179f: Status 404 returned error can't find the container with id bed27599fb721e2e6ff4c9d3a2c409d5288186503aa29b9cd938372e4ba8179f
Nov 02 15:23:30.224236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:30.224215    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"]
Nov 02 15:23:30.294684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:30.293270    7019 patch_prober.go:29] interesting pod/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 15:23:30.294684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:30.293331    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 15:23:30.486726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:30.478169    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:30.713512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:30.713468    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:30.954074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:30.954030    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg" event={"ID":"14d47b64-646f-4d92-9c08-1b94841aaba4","Type":"ContainerStarted","Data":"2d19a6a0f057dcec59309baf798bfd882a669a724a3b6c8ae31c4ee5599f140b"}
Nov 02 15:23:30.954379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:30.954098    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg" event={"ID":"14d47b64-646f-4d92-9c08-1b94841aaba4","Type":"ContainerStarted","Data":"bed27599fb721e2e6ff4c9d3a2c409d5288186503aa29b9cd938372e4ba8179f"}
Nov 02 15:23:31.023454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.023396    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg" podStartSLOduration=2.023378016 podStartE2EDuration="2.023378016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:31.021495467 +0000 UTC m=+1605.275449092" watchObservedRunningTime="2026-11-02 15:23:31.023378016 +0000 UTC m=+1605.277331641"
Nov 02 15:23:31.102248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.102210    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:31.257540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.257458    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:31.286670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.284370    7019 patch_prober.go:29] interesting pod/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 15:23:31.286670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.284431    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 15:23:31.438894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.438848    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"]
Nov 02 15:23:31.439148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.439093    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy" containerID="cri-o://5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd" gracePeriod=30
Nov 02 15:23:31.444323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.444291    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"]
Nov 02 15:23:31.444531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.444504    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy" containerID="cri-o://4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20" gracePeriod=30
Nov 02 15:23:31.450494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.450474    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"]
Nov 02 15:23:31.451517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.451489    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy" containerID="cri-o://5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22" gracePeriod=30
Nov 02 15:23:31.598726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.598381    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:31.966113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.966076    7019 generic.go:356] "Generic (PLEG): container finished" podID="14d47b64-646f-4d92-9c08-1b94841aaba4" containerID="2d19a6a0f057dcec59309baf798bfd882a669a724a3b6c8ae31c4ee5599f140b" exitCode=0
Nov 02 15:23:31.966373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:31.966351    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg" event={"ID":"14d47b64-646f-4d92-9c08-1b94841aaba4","Type":"ContainerDied","Data":"2d19a6a0f057dcec59309baf798bfd882a669a724a3b6c8ae31c4ee5599f140b"}
Nov 02 15:23:32.124724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:32.124685    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:32.293742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:32.290692    7019 patch_prober.go:29] interesting pod/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 15:23:32.293742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:32.290879    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 15:23:32.597995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:32.597901    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:32.750019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:32.749981    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:33.297672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.297017    7019 patch_prober.go:29] interesting pod/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 15:23:33.297672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.297078    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 15:23:33.451929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.451865    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:33.610170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.610135    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d47b64-646f-4d92-9c08-1b94841aaba4-kube-api-access-9wvb5\" (UniqueName: \"kubernetes.io/projected/14d47b64-646f-4d92-9c08-1b94841aaba4-kube-api-access-9wvb5\") pod \"14d47b64-646f-4d92-9c08-1b94841aaba4\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") "
Nov 02 15:23:33.610391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.610218    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-openshift-ca\") pod \"14d47b64-646f-4d92-9c08-1b94841aaba4\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") "
Nov 02 15:23:33.610391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.610380    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-xks-ca\") pod \"14d47b64-646f-4d92-9c08-1b94841aaba4\" (UID: \"14d47b64-646f-4d92-9c08-1b94841aaba4\") "
Nov 02 15:23:33.610630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.610602    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-openshift-ca" pod "14d47b64-646f-4d92-9c08-1b94841aaba4" (UID: "14d47b64-646f-4d92-9c08-1b94841aaba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:33.611041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.611020    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:33.612616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.612540    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-xks-ca" pod "14d47b64-646f-4d92-9c08-1b94841aaba4" (UID: "14d47b64-646f-4d92-9c08-1b94841aaba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:33.612902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.612878    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d47b64-646f-4d92-9c08-1b94841aaba4-kube-api-access-9wvb5" pod "14d47b64-646f-4d92-9c08-1b94841aaba4" (UID: "14d47b64-646f-4d92-9c08-1b94841aaba4"). InnerVolumeSpecName "kube-api-access-9wvb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:33.712948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.712843    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvb5\" (UniqueName: \"kubernetes.io/projected/14d47b64-646f-4d92-9c08-1b94841aaba4-kube-api-access-9wvb5\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:33.712948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:33.712878    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14d47b64-646f-4d92-9c08-1b94841aaba4-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:34.213273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:34.213231    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg" event={"ID":"14d47b64-646f-4d92-9c08-1b94841aaba4","Type":"ContainerDied","Data":"bed27599fb721e2e6ff4c9d3a2c409d5288186503aa29b9cd938372e4ba8179f"}
Nov 02 15:23:34.213478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:34.213293    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed27599fb721e2e6ff4c9d3a2c409d5288186503aa29b9cd938372e4ba8179f"
Nov 02 15:23:34.213478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:34.213245    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"
Nov 02 15:23:34.288669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:34.286749    7019 patch_prober.go:29] interesting pod/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 15:23:34.288669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:34.286812    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 15:23:34.303670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:34.298906    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:35.290212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:35.290167    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:23:35.290603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:35.290576    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:23:35.291440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:35.291418    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:23:36.114718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.114680    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:36.655326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.655306    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:23:36.664038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.664010    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-kube-api-access-cx5xj\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-kube-api-access-cx5xj\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.670794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.664788    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-socket\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.671077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.671055    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-podinfo\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.671208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.671195    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-envoy\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.671400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.671387    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-token\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-token\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.671571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.671557    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-certs\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.671737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.671721    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istiod-ca-cert\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.671883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.671870    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-data\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.672002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.671987    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-credential-socket\") pod \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\" (UID: \"771bdc54-87f5-4666-a7b1-bb77ad2197f8\") "
Nov 02 15:23:36.673371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.665430    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-socket" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.673371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.672363    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-certs" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.673371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.673165    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-credential-socket" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.673371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.673181    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.673371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.673224    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.673371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.673301    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istiod-ca-cert" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:36.673371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.673353    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-data" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.677892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.677869    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-token" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.679077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.679019    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-podinfo" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:36.684153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.684113    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-envoy" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.704721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.704635    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-kube-api-access-cx5xj" pod "771bdc54-87f5-4666-a7b1-bb77ad2197f8" (UID: "771bdc54-87f5-4666-a7b1-bb77ad2197f8"). InnerVolumeSpecName "kube-api-access-cx5xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.783468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.783429    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.783729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.783714    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.783825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.783812    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.783925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.783911    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx5xj\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-kube-api-access-cx5xj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.784015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.784004    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.784107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.784097    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.784184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.784175    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771bdc54-87f5-4666-a7b1-bb77ad2197f8-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.816728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.816627    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:23:36.838801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.838772    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:23:36.884794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.884763    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-envoy\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.884986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.884837    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istiod-ca-cert\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.884986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.884865    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-envoy\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.884986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.884962    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istiod-ca-cert\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.885162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885025    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-credential-socket\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.885162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885085    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-kube-api-access-896jc\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-kube-api-access-896jc\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.885162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885154    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-token\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.885317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885214    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-socket\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.885317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885263    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-podinfo\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.885421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885315    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-data\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.885421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885358    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-certs\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.885421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885400    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-certs\") pod \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\" (UID: \"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3\") "
Nov 02 15:23:36.885573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885471    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-credential-socket" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.885573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885479    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-kube-api-access-snwzt\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-kube-api-access-snwzt\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.885573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885511    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-credential-socket\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.885751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885637    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-podinfo\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.885751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885725    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-token\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-token\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.885854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885763    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-data\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.885854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885764    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istiod-ca-cert" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:36.885854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.885798    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-socket\") pod \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\" (UID: \"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0\") "
Nov 02 15:23:36.886045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886004    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-certs" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.886203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886180    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-socket" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.886377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886360    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-certs" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.886926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886902    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.887036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886931    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.887036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886946    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.887036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886962    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.887036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886977    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.887036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.886910    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-socket" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.887382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.887360    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-data" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.887382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.887376    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istiod-ca-cert" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:36.888503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.888477    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-credential-socket" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.888846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.888816    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-data" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.893699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.893502    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-token" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.894607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.894555    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-podinfo" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:36.894918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.894885    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-kube-api-access-896jc" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "kube-api-access-896jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.895886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.895793    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-token" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.896038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.895861    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-envoy" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.896140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.895886    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-envoy" pod "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" (UID: "9eaf17e3-6c7b-4dc7-a76d-ba88934916e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:36.897572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.897524    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-podinfo" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:36.897992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.897971    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-kube-api-access-snwzt" pod "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" (UID: "ae4a3628-bdc9-4e16-8eb8-a46dd09572b0"). InnerVolumeSpecName "kube-api-access-snwzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:36.989261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989213    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989253    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snwzt\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-kube-api-access-snwzt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989268    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989282    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989296    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989308    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989319    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989331    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989343    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989355    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896jc\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-kube-api-access-896jc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989367    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989381    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.989604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:36.989392    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:37.238595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.238564    7019 generic.go:356] "Generic (PLEG): container finished" podID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerID="4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20" exitCode=0
Nov 02 15:23:37.239096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.238610    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"
Nov 02 15:23:37.239096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.238695    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" event={"ID":"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0","Type":"ContainerDied","Data":"4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20"}
Nov 02 15:23:37.239096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.238742    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58" event={"ID":"ae4a3628-bdc9-4e16-8eb8-a46dd09572b0","Type":"ContainerDied","Data":"3871b1595b02a6339ec9e044a903d0718aec3116817f79cf724f8937d3692e59"}
Nov 02 15:23:37.239096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.238766    7019 scope.go:122] "RemoveContainer" containerID="4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20"
Nov 02 15:23:37.240224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.240193    7019 generic.go:356] "Generic (PLEG): container finished" podID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerID="5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd" exitCode=0
Nov 02 15:23:37.240333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.240240    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" event={"ID":"771bdc54-87f5-4666-a7b1-bb77ad2197f8","Type":"ContainerDied","Data":"5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd"}
Nov 02 15:23:37.240333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.240294    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb" event={"ID":"771bdc54-87f5-4666-a7b1-bb77ad2197f8","Type":"ContainerDied","Data":"cd04b08f0bbfbb5f91f616d7ef98604db5e01787fbe6351e19b682c4a7ec56ad"}
Nov 02 15:23:37.240333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.240251    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"
Nov 02 15:23:37.241636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.241616    7019 generic.go:356] "Generic (PLEG): container finished" podID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerID="5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22" exitCode=0
Nov 02 15:23:37.241768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.241678    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"
Nov 02 15:23:37.241768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.241690    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" event={"ID":"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3","Type":"ContainerDied","Data":"5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22"}
Nov 02 15:23:37.241869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.241767    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz" event={"ID":"9eaf17e3-6c7b-4dc7-a76d-ba88934916e3","Type":"ContainerDied","Data":"eadebf56c6f894100e03ca455fe9ba460671caff206d7d9267250987564e3e31"}
Nov 02 15:23:37.254123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.254062    7019 scope.go:122] "RemoveContainer" containerID="4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20"
Nov 02 15:23:37.254551 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:23:37.254531    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20\": container with ID starting with 4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20 not found: ID does not exist" containerID="4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20"
Nov 02 15:23:37.254669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.254559    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20"} err="failed to get container status \"4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20\": rpc error: code = NotFound desc = could not find container \"4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20\": container with ID starting with 4821ee263aa0b2ffa517f10417c376d30284c4da459759f0a5713a5c7febbb20 not found: ID does not exist"
Nov 02 15:23:37.254669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.254577    7019 scope.go:122] "RemoveContainer" containerID="5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd"
Nov 02 15:23:37.264989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.264973    7019 scope.go:122] "RemoveContainer" containerID="5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd"
Nov 02 15:23:37.265322 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:23:37.265301    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd\": container with ID starting with 5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd not found: ID does not exist" containerID="5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd"
Nov 02 15:23:37.265407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.265328    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd"} err="failed to get container status \"5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd\": rpc error: code = NotFound desc = could not find container \"5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd\": container with ID starting with 5533e70031ae222f81ad2729f1ae4029668055463aeb8392a9729181a10053fd not found: ID does not exist"
Nov 02 15:23:37.265407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.265345    7019 scope.go:122] "RemoveContainer" containerID="5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22"
Nov 02 15:23:37.277545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.277526    7019 scope.go:122] "RemoveContainer" containerID="5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22"
Nov 02 15:23:37.277874 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:23:37.277845    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22\": container with ID starting with 5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22 not found: ID does not exist" containerID="5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22"
Nov 02 15:23:37.277961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.277882    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22"} err="failed to get container status \"5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22\": rpc error: code = NotFound desc = could not find container \"5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22\": container with ID starting with 5001e728643e76690557e834e799665e1d9eeb64a35330006aede79d301b7f22 not found: ID does not exist"
Nov 02 15:23:37.331165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.331135    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"]
Nov 02 15:23:37.335162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.335137    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-zdx58"]
Nov 02 15:23:37.385599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.385561    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"]
Nov 02 15:23:37.392622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.392592    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-m7cpb"]
Nov 02 15:23:37.433450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.433417    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"]
Nov 02 15:23:37.440942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.440917    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d7df16e-openshift-default-57b7fc5b7c-g27rz"]
Nov 02 15:23:37.865455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.865423    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" path="/var/lib/kubelet/pods/771bdc54-87f5-4666-a7b1-bb77ad2197f8/volumes"
Nov 02 15:23:37.866085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.866029    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" path="/var/lib/kubelet/pods/9eaf17e3-6c7b-4dc7-a76d-ba88934916e3/volumes"
Nov 02 15:23:37.866867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.866661    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" path="/var/lib/kubelet/pods/ae4a3628-bdc9-4e16-8eb8-a46dd09572b0/volumes"
Nov 02 15:23:37.869508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:37.869486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:38.402215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:38.401200    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:38.561204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:38.561155    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:38.565028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:38.564999    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"]
Nov 02 15:23:38.568956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:38.568929    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55e4c3-mfd2z"]
Nov 02 15:23:38.729229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:38.729133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:39.227060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:39.227030    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:39.863950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:39.863918    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad724f70-ce95-4ae7-b751-74261b3b959d" path="/var/lib/kubelet/pods/ad724f70-ce95-4ae7-b751-74261b3b959d/volumes"
Nov 02 15:23:40.084836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:40.084807    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:40.235371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:40.235277    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:40.685352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:40.685311    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:42.569757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:42.569715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:44.302777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:44.302731    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:45.970972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:45.970935    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:46.169495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:46.169457    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:46.351463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:46.351380    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"]
Nov 02 15:23:46.351702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:46.351679    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" containerID="cri-o://0f0a54b7c2fa8eeae09e66a574b0763f44dc35b7a1a7145cef4fb498cd6893f8" gracePeriod=30
Nov 02 15:23:46.435593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:46.435538    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:47.012633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.012598    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"]
Nov 02 15:23:47.013516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013494    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy"
Nov 02 15:23:47.013516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013516    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy"
Nov 02 15:23:47.013679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013547    7019 cpu_manager.go:414] "containerMap: removing container" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy"
Nov 02 15:23:47.013679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013553    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy"
Nov 02 15:23:47.013679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013558    7019 cpu_manager.go:414] "containerMap: removing container" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy"
Nov 02 15:23:47.013679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013563    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy"
Nov 02 15:23:47.013679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013570    7019 cpu_manager.go:414] "containerMap: removing container" podUID="14d47b64-646f-4d92-9c08-1b94841aaba4" containerName="revoke-keys"
Nov 02 15:23:47.013679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013574    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d47b64-646f-4d92-9c08-1b94841aaba4" containerName="revoke-keys"
Nov 02 15:23:47.013869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013699    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="771bdc54-87f5-4666-a7b1-bb77ad2197f8" containerName="istio-proxy"
Nov 02 15:23:47.013869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013706    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eaf17e3-6c7b-4dc7-a76d-ba88934916e3" containerName="istio-proxy"
Nov 02 15:23:47.013869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013714    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d47b64-646f-4d92-9c08-1b94841aaba4" containerName="revoke-keys"
Nov 02 15:23:47.013869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.013728    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4a3628-bdc9-4e16-8eb8-a46dd09572b0" containerName="istio-proxy"
Nov 02 15:23:47.017330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.017313    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.041487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.041460    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"]
Nov 02 15:23:47.100213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.100172    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:47.119854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.119812    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmrj\" (UniqueName: \"kubernetes.io/projected/e55bce1c-05d1-4992-a0de-60232fe924ad-kube-api-access-fhmrj\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.120021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.119998    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.120089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.120067    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.120296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.120275    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:23:47.120610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.120593    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:47.222129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.222097    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.222300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.222148    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.222384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.222364    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmrj\" (UniqueName: \"kubernetes.io/projected/e55bce1c-05d1-4992-a0de-60232fe924ad-kube-api-access-fhmrj\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.222792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.222767    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.224527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.224509    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.247608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.247585    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmrj\" (UniqueName: \"kubernetes.io/projected/e55bce1c-05d1-4992-a0de-60232fe924ad-kube-api-access-fhmrj\") pod \"maas-api-revoke-keys-e2e-api-b-815746-gn92t\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.332151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.332077    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:47.645618 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:23:47.645586    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55bce1c_05d1_4992_a0de_60232fe924ad.slice/crio-73d3166f83d5bdb493b07ba5593af3590362e63db0be574dec2504dcb27583fe WatchSource:0}: Error finding container 73d3166f83d5bdb493b07ba5593af3590362e63db0be574dec2504dcb27583fe: Status 404 returned error can't find the container with id 73d3166f83d5bdb493b07ba5593af3590362e63db0be574dec2504dcb27583fe
Nov 02 15:23:47.646169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.646132    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"]
Nov 02 15:23:47.719472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:47.719437    7019 scope.go:122] "RemoveContainer" containerID="b38886e083995b4c8dfddc2f031edb8521fc454b182cd7b80e6b451d82741d4e"
Nov 02 15:23:48.088564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:48.088527    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:48.270680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:48.270629    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:48.414421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:48.414387    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:48.580762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:48.580718    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t" event={"ID":"e55bce1c-05d1-4992-a0de-60232fe924ad","Type":"ContainerStarted","Data":"9e369823aa7178e38b66b4e83bf01f1dba0e00d09a6f83aa356b2602ef646375"}
Nov 02 15:23:48.580936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:48.580786    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t" event={"ID":"e55bce1c-05d1-4992-a0de-60232fe924ad","Type":"ContainerStarted","Data":"73d3166f83d5bdb493b07ba5593af3590362e63db0be574dec2504dcb27583fe"}
Nov 02 15:23:48.627900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:48.627842    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t" podStartSLOduration=2.627827462 podStartE2EDuration="2.627827462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:48.625926189 +0000 UTC m=+1622.879879811" watchObservedRunningTime="2026-11-02 15:23:48.627827462 +0000 UTC m=+1622.881781086"
Nov 02 15:23:49.482446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:49.482408    7019 patch_prober.go:29] interesting pod/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:23:49.482955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:49.482476    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:23:49.591292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:49.591249    7019 generic.go:356] "Generic (PLEG): container finished" podID="e55bce1c-05d1-4992-a0de-60232fe924ad" containerID="9e369823aa7178e38b66b4e83bf01f1dba0e00d09a6f83aa356b2602ef646375" exitCode=0
Nov 02 15:23:49.591470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:49.591335    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t" event={"ID":"e55bce1c-05d1-4992-a0de-60232fe924ad","Type":"ContainerDied","Data":"9e369823aa7178e38b66b4e83bf01f1dba0e00d09a6f83aa356b2602ef646375"}
Nov 02 15:23:50.167679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.158488    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:50.759149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.759121    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:50.782466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.782437    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-xks-ca\") pod \"e55bce1c-05d1-4992-a0de-60232fe924ad\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") "
Nov 02 15:23:50.782622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.782524    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-openshift-ca\") pod \"e55bce1c-05d1-4992-a0de-60232fe924ad\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") "
Nov 02 15:23:50.782622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.782565    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55bce1c-05d1-4992-a0de-60232fe924ad-kube-api-access-fhmrj\" (UniqueName: \"kubernetes.io/projected/e55bce1c-05d1-4992-a0de-60232fe924ad-kube-api-access-fhmrj\") pod \"e55bce1c-05d1-4992-a0de-60232fe924ad\" (UID: \"e55bce1c-05d1-4992-a0de-60232fe924ad\") "
Nov 02 15:23:50.782911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.782885    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-openshift-ca" pod "e55bce1c-05d1-4992-a0de-60232fe924ad" (UID: "e55bce1c-05d1-4992-a0de-60232fe924ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:50.783463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.783433    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:50.784800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.784774    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-xks-ca" pod "e55bce1c-05d1-4992-a0de-60232fe924ad" (UID: "e55bce1c-05d1-4992-a0de-60232fe924ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:50.784879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.784778    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55bce1c-05d1-4992-a0de-60232fe924ad-kube-api-access-fhmrj" pod "e55bce1c-05d1-4992-a0de-60232fe924ad" (UID: "e55bce1c-05d1-4992-a0de-60232fe924ad"). InnerVolumeSpecName "kube-api-access-fhmrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:50.885408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.885377    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55bce1c-05d1-4992-a0de-60232fe924ad-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:50.885408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:50.885406    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhmrj\" (UniqueName: \"kubernetes.io/projected/e55bce1c-05d1-4992-a0de-60232fe924ad-kube-api-access-fhmrj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.610107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.610075    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"
Nov 02 15:23:51.610296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.610107    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t" event={"ID":"e55bce1c-05d1-4992-a0de-60232fe924ad","Type":"ContainerDied","Data":"73d3166f83d5bdb493b07ba5593af3590362e63db0be574dec2504dcb27583fe"}
Nov 02 15:23:51.610296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.610164    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73d3166f83d5bdb493b07ba5593af3590362e63db0be574dec2504dcb27583fe"
Nov 02 15:23:51.612380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.612359    7019 generic.go:356] "Generic (PLEG): container finished" podID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerID="0f0a54b7c2fa8eeae09e66a574b0763f44dc35b7a1a7145cef4fb498cd6893f8" exitCode=0
Nov 02 15:23:51.612517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.612413    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" event={"ID":"e7c27a52-60bb-46a5-b431-0d231bfaf36a","Type":"ContainerDied","Data":"0f0a54b7c2fa8eeae09e66a574b0763f44dc35b7a1a7145cef4fb498cd6893f8"}
Nov 02 15:23:51.612517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.612451    7019 scope.go:122] "RemoveContainer" containerID="ba743bb9b6c91e91893ff6ac1bf55fa01653c4c533cc64af65bb7f1a46219112"
Nov 02 15:23:51.636818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.636791    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:23:51.697682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.697612    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-credential-socket\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.697869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.697783    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-envoy\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.697934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.697887    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istiod-ca-cert\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.697994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.697958    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-socket\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.698096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.698079    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-kube-api-access-vs4j6\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-kube-api-access-vs4j6\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.698152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.698121    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-data\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.698217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.698203    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-certs\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.698313 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.698259    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-podinfo\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.698376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.698318    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-token\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-token\") pod \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\" (UID: \"e7c27a52-60bb-46a5-b431-0d231bfaf36a\") "
Nov 02 15:23:51.699798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.699564    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-socket" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:51.699798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.699681    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-data" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:51.699975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.699885    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istiod-ca-cert" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:51.700168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.700131    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-certs" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:51.700913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.700885    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-credential-socket" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:51.701594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.701573    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-token" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:51.701714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.701597    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-envoy" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:23:51.702071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.702055    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-kube-api-access-vs4j6" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "kube-api-access-vs4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:51.702511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.702492    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-podinfo" pod "e7c27a52-60bb-46a5-b431-0d231bfaf36a" (UID: "e7c27a52-60bb-46a5-b431-0d231bfaf36a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:23:51.800045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800014    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800040    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800049    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4j6\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-kube-api-access-vs4j6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800058    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800066    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800074    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800082    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800090    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:51.800486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:51.800098    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c27a52-60bb-46a5-b431-0d231bfaf36a-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:52.087109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:52.087017    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:52.622945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:52.622914    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"
Nov 02 15:23:52.622945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:52.622935    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr" event={"ID":"e7c27a52-60bb-46a5-b431-0d231bfaf36a","Type":"ContainerDied","Data":"2108832fc27bc4c7b3584900b901f2373651c0752ff3417fe9432fcb8580f5a4"}
Nov 02 15:23:52.623208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:52.623014    7019 scope.go:122] "RemoveContainer" containerID="0f0a54b7c2fa8eeae09e66a574b0763f44dc35b7a1a7145cef4fb498cd6893f8"
Nov 02 15:23:52.683600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:52.683568    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"]
Nov 02 15:23:52.697185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:52.697149    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-55e4c3-openshift-default-6dbbbf7bb7-nlscr"]
Nov 02 15:23:52.711312 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:23:52.711279    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:23:53.033733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:53.033633    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"]
Nov 02 15:23:53.036913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:53.036888    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f043844-xj7c4"]
Nov 02 15:23:53.715843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:53.715809    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:53.864878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:53.864844    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54a83c3-0a90-4813-8a26-6a873151483b" path="/var/lib/kubelet/pods/b54a83c3-0a90-4813-8a26-6a873151483b/volumes"
Nov 02 15:23:53.865577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:53.865558    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" path="/var/lib/kubelet/pods/e7c27a52-60bb-46a5-b431-0d231bfaf36a/volumes"
Nov 02 15:23:55.288390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:55.288354    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:56.852643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.852559    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"]
Nov 02 15:23:56.853479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.853456    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e55bce1c-05d1-4992-a0de-60232fe924ad" containerName="revoke-keys"
Nov 02 15:23:56.853595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.853479    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55bce1c-05d1-4992-a0de-60232fe924ad" containerName="revoke-keys"
Nov 02 15:23:56.853595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.853500    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy"
Nov 02 15:23:56.853595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.853507    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy"
Nov 02 15:23:56.853804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.853748    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy"
Nov 02 15:23:56.853804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.853777    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55bce1c-05d1-4992-a0de-60232fe924ad" containerName="revoke-keys"
Nov 02 15:23:56.857672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.857624    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:56.879691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.878464    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"]
Nov 02 15:23:56.886304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.886279    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:56.886925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.886369    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:23:56.987299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.987266    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:56.987472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.987324    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k59f\" (UniqueName: \"kubernetes.io/projected/73f893f0-b13d-4c4d-8be1-4f750ca074ff-kube-api-access-4k59f\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:56.987526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:56.987463    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.075759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.075723    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:57.099315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.099269    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.100418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.100392    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k59f\" (UniqueName: \"kubernetes.io/projected/73f893f0-b13d-4c4d-8be1-4f750ca074ff-kube-api-access-4k59f\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.100640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.100622    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.101292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.100313    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.129495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.129461    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.130836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.130806    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k59f\" (UniqueName: \"kubernetes.io/projected/73f893f0-b13d-4c4d-8be1-4f750ca074ff-kube-api-access-4k59f\") pod \"maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.171841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.171807    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:57.482938 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:23:57.482904    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f893f0_b13d_4c4d_8be1_4f750ca074ff.slice/crio-8f5a0963e9143738ffdfe9a8676aa91da698ffaaca2bb195bac1b1aa161d1d58 WatchSource:0}: Error finding container 8f5a0963e9143738ffdfe9a8676aa91da698ffaaca2bb195bac1b1aa161d1d58: Status 404 returned error can't find the container with id 8f5a0963e9143738ffdfe9a8676aa91da698ffaaca2bb195bac1b1aa161d1d58
Nov 02 15:23:57.484047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.484025    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"]
Nov 02 15:23:57.664858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.664819    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w" event={"ID":"73f893f0-b13d-4c4d-8be1-4f750ca074ff","Type":"ContainerStarted","Data":"8f5a0963e9143738ffdfe9a8676aa91da698ffaaca2bb195bac1b1aa161d1d58"}
Nov 02 15:23:57.902491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:57.902454    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:58.102589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:58.102510    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:58.676223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:58.676187    7019 generic.go:356] "Generic (PLEG): container finished" podID="73f893f0-b13d-4c4d-8be1-4f750ca074ff" containerID="1c2ab8a02d8d77210531a12ef8ab5f2c2bb1fb7652039a0b48c73592137ee4e6" exitCode=0
Nov 02 15:23:58.676388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:58.676264    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w" event={"ID":"73f893f0-b13d-4c4d-8be1-4f750ca074ff","Type":"ContainerDied","Data":"1c2ab8a02d8d77210531a12ef8ab5f2c2bb1fb7652039a0b48c73592137ee4e6"}
Nov 02 15:23:58.999677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:58.999498    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:23:59.864281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.864252    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:23:59.963984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.963899    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f893f0-b13d-4c4d-8be1-4f750ca074ff-kube-api-access-4k59f\" (UniqueName: \"kubernetes.io/projected/73f893f0-b13d-4c4d-8be1-4f750ca074ff-kube-api-access-4k59f\") pod \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") "
Nov 02 15:23:59.963984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.963968    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-xks-ca\") pod \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") "
Nov 02 15:23:59.964219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.964063    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-openshift-ca\") pod \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\" (UID: \"73f893f0-b13d-4c4d-8be1-4f750ca074ff\") "
Nov 02 15:23:59.964430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.964392    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-openshift-ca" pod "73f893f0-b13d-4c4d-8be1-4f750ca074ff" (UID: "73f893f0-b13d-4c4d-8be1-4f750ca074ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:59.969475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.964633    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:23:59.969475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.966162    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-xks-ca" pod "73f893f0-b13d-4c4d-8be1-4f750ca074ff" (UID: "73f893f0-b13d-4c4d-8be1-4f750ca074ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:59.969475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:23:59.967754    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f893f0-b13d-4c4d-8be1-4f750ca074ff-kube-api-access-4k59f" pod "73f893f0-b13d-4c4d-8be1-4f750ca074ff" (UID: "73f893f0-b13d-4c4d-8be1-4f750ca074ff"). InnerVolumeSpecName "kube-api-access-4k59f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:24:00.066019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:00.065982    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k59f\" (UniqueName: \"kubernetes.io/projected/73f893f0-b13d-4c4d-8be1-4f750ca074ff-kube-api-access-4k59f\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:24:00.066019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:00.066019    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73f893f0-b13d-4c4d-8be1-4f750ca074ff-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:24:00.691422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:00.691373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:00.698417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:00.698389    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"
Nov 02 15:24:00.698417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:00.698401    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w" event={"ID":"73f893f0-b13d-4c4d-8be1-4f750ca074ff","Type":"ContainerDied","Data":"8f5a0963e9143738ffdfe9a8676aa91da698ffaaca2bb195bac1b1aa161d1d58"}
Nov 02 15:24:00.698627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:00.698494    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f5a0963e9143738ffdfe9a8676aa91da698ffaaca2bb195bac1b1aa161d1d58"
Nov 02 15:24:00.835601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:00.835560    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:02.266415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:02.266376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:03.879019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:03.878983    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:05.484512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:05.484481    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:07.211218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:07.211178    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:07.621608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:07.621568    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:07.780772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:07.780716    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:09.012274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:09.012235    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:09.739670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:09.732830    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:24:10.410459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:10.410417    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:10.662625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:10.658812    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:12.832575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:12.832538    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:13.738686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:13.737754    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:13.739669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:13.739014    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:13.825828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:13.825768    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:13.851634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:13.851587    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:14.011519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:14.011409    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:14.177372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:14.016480    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:14.278942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:14.278806    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw" podUID="3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.120:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:14.332674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:14.325750    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:24:14.935347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:14.935297    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:16.688068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:16.688034    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:16.937194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:16.937158    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:18.514531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:18.514455    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:20.297354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:20.297261    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:21.728962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:21.728917    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:21.933111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:21.933011    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:22.089546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:22.089434    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:22.091120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:22.091091    7019 generic.go:356] "Generic (PLEG): container finished" podID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerID="1a71d7181042133f56566a3eebc3b3ba503559008119cafb0993a645a0cad232" exitCode=0
Nov 02 15:24:22.091238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:22.091165    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" event={"ID":"fa84d2fd-0959-4ef0-baf5-d59c36f24337","Type":"ContainerDied","Data":"1a71d7181042133f56566a3eebc3b3ba503559008119cafb0993a645a0cad232"}
Nov 02 15:24:22.091682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:22.091633    7019 scope.go:122] "RemoveContainer" containerID="1a71d7181042133f56566a3eebc3b3ba503559008119cafb0993a645a0cad232"
Nov 02 15:24:22.093928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:22.093893    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:24:22.094015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:22.093986    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:24:22.094060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:22.094036    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:24:22.707976 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:24:22.707933    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:24:23.104708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:23.104596    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" event={"ID":"fa84d2fd-0959-4ef0-baf5-d59c36f24337","Type":"ContainerStarted","Data":"4442e293b635cfb1dfd20386a7f9d3165007c729b42a2f22368e146db7f6a02e"}
Nov 02 15:24:23.448418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:23.448386    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:24:23.448602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:23.448564    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 15:24:23.448726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:23.448619    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 15:24:23.615673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:23.610840    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:24.448855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:24.448818    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 15:24:24.449357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:24.448905    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 15:24:25.185352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:25.185319    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:25.448719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:25.448585    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 15:24:25.448719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:25.448704    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 15:24:26.449107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:26.449053    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 15:24:26.449521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:26.449136    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 15:24:26.800551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:26.800465    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:27.449038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:27.449005    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 15:24:27.449217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:27.449069    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 15:24:28.448261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:28.448231    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 15:24:28.448435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:28.448280    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 15:24:28.487672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:28.486672    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:29.412924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:29.412894    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:24:29.453477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:29.453450    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:24:30.005823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:30.005784    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:30.160295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:30.160269    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:24:30.429336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:30.429300    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:30.606490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:30.606455    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:31.835668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:31.834893    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:33.605792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:33.605755    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:35.546702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.544562    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:35.879192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.879157    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"]
Nov 02 15:24:35.880137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.880112    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy"
Nov 02 15:24:35.880137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.880135    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy"
Nov 02 15:24:35.880285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.880160    7019 cpu_manager.go:414] "containerMap: removing container" podUID="73f893f0-b13d-4c4d-8be1-4f750ca074ff" containerName="revoke-keys"
Nov 02 15:24:35.880285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.880166    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f893f0-b13d-4c4d-8be1-4f750ca074ff" containerName="revoke-keys"
Nov 02 15:24:35.880285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.880281    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f893f0-b13d-4c4d-8be1-4f750ca074ff" containerName="revoke-keys"
Nov 02 15:24:35.880385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.880288    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c27a52-60bb-46a5-b431-0d231bfaf36a" containerName="istio-proxy"
Nov 02 15:24:35.883407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.883386    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:35.901631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.901608    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a713aa-openshift-default-dockercfg-vrt2d\""
Nov 02 15:24:35.922151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:35.922115    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"]
Nov 02 15:24:36.032019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.031980    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b035673-1a9d-4150-956f-8c1512f0d857-istio-podinfo\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.032019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.032020    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-credential-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.032247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.032048    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6px2\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-kube-api-access-n6px2\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.032247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.032206    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-certs\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.032333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.032260    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.032538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.032513    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b035673-1a9d-4150-956f-8c1512f0d857-istiod-ca-cert\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.032587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.032559    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-istio-token\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.032850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.032823    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-envoy\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.033110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.033087    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-data\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.134712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.134603    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6px2\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-kube-api-access-n6px2\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.134869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.134715    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-certs\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.134869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.134774    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.134964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.134878    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b035673-1a9d-4150-956f-8c1512f0d857-istiod-ca-cert\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.134964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.134907    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-istio-token\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135046    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-envoy\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135129    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-data\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135167    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135128    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-certs\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135247    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b035673-1a9d-4150-956f-8c1512f0d857-istio-podinfo\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135284    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-credential-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135356    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-data\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135589    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b035673-1a9d-4150-956f-8c1512f0d857-istiod-ca-cert\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.135710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.135619    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-credential-socket\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.137438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.137412    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-envoy\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.137598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.137577    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b035673-1a9d-4150-956f-8c1512f0d857-istio-podinfo\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.160279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.160248    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-istio-token\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.160389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.160314    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6px2\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-kube-api-access-n6px2\") pod \"e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") " pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.197605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.197575    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:36.529613 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:24:36.529582    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b035673_1a9d_4150_956f_8c1512f0d857.slice/crio-e4ac393d6728ea7b7068f3bd46d9f859f5112878b4e4f470fe9162c34f3bd2d3 WatchSource:0}: Error finding container e4ac393d6728ea7b7068f3bd46d9f859f5112878b4e4f470fe9162c34f3bd2d3: Status 404 returned error can't find the container with id e4ac393d6728ea7b7068f3bd46d9f859f5112878b4e4f470fe9162c34f3bd2d3
Nov 02 15:24:36.531050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.531026    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"]
Nov 02 15:24:36.532774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.532738    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:24:36.532856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.532806    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:24:36.532856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:36.532836    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:24:37.209740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:37.209702    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:37.224861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:37.224828    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" event={"ID":"3b035673-1a9d-4150-956f-8c1512f0d857","Type":"ContainerStarted","Data":"73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493"}
Nov 02 15:24:37.225005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:37.224893    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" event={"ID":"3b035673-1a9d-4150-956f-8c1512f0d857","Type":"ContainerStarted","Data":"e4ac393d6728ea7b7068f3bd46d9f859f5112878b4e4f470fe9162c34f3bd2d3"}
Nov 02 15:24:37.316415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:37.313190    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" podStartSLOduration=2.3131619 podStartE2EDuration="2.3131619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:37.309176359 +0000 UTC m=+1671.563130007" watchObservedRunningTime="2026-11-02 15:24:37.3131619 +0000 UTC m=+1671.567115543"
Nov 02 15:24:37.399540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:37.399506    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:38.197967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:38.197929    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:38.200019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:38.199988    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:24:38.200325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:38.200288    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 15:24:39.198131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:39.198090    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:24:39.198742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:39.198710    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 15:24:39.214669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:39.213147    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:40.198668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:40.198396    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:24:40.198668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:40.198485    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 15:24:40.280592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:40.280557    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:40.945050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:40.944992    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:41.198738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:41.198271    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:24:41.198738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:41.198358    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 15:24:42.198377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:42.198346    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Nov 02 15:24:42.198625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:42.198600    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Nov 02 15:24:42.599531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:42.599421    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:43.203468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:43.203426    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:43.203834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:43.203814    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:43.204537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:43.204513    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:24:44.292703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:44.292639    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:45.822197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:45.821349    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:47.207678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:47.207603    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:47.579128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:47.579035    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:47.775105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:47.775069    7019 scope.go:122] "RemoveContainer" containerID="082b52080ca884cc4d0ce8065abc938e5629d4af2a5503a49c3ccd9a18eb3b61"
Nov 02 15:24:49.224312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:49.222299    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:50.586163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:50.586127    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:51.270009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:51.269974    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:51.377906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:51.377872    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"]
Nov 02 15:24:51.384426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:51.384393    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1cdbe6-4dhrg"]
Nov 02 15:24:51.457306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:51.457271    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:51.863158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:51.863127    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d47b64-646f-4d92-9c08-1b94841aaba4" path="/var/lib/kubelet/pods/14d47b64-646f-4d92-9c08-1b94841aaba4/volumes"
Nov 02 15:24:51.923803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:51.923761    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:52.160735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:52.160597    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:52.340537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:52.340506    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:52.707956 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:24:52.707915    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:24:53.286620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:53.286584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:54.954109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:54.954072    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:55.036143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:55.036109    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"]
Nov 02 15:24:55.048434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:55.048402    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd06887a-tb46l"]
Nov 02 15:24:55.863434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:55.863399    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874824ca-f99e-453d-8eee-58c708804f96" path="/var/lib/kubelet/pods/874824ca-f99e-453d-8eee-58c708804f96/volumes"
Nov 02 15:24:56.621255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:56.621223    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:58.282211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:58.282173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:59.027226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:59.027183    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:59.280856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:59.280761    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:24:59.388438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:59.388402    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"]
Nov 02 15:24:59.388956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:59.388811    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy" containerID="cri-o://141095b0ceff7217a7759f6aad807129ff8310b7ee858f1103560f1e7c19f5c9" gracePeriod=30
Nov 02 15:24:59.444277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:24:59.443823    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:00.294284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.294229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:00.440197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.440162    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"]
Nov 02 15:25:00.447252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.447231    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.466007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.465971    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:25:00.466170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.466094    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:25:00.468280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.468218    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"]
Nov 02 15:25:00.593718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.593593    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lhkp\" (UniqueName: \"kubernetes.io/projected/305f777f-960b-432b-b5a5-191baaa576de-kube-api-access-7lhkp\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.593718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.593663    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f777f-960b-432b-b5a5-191baaa576de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.593975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.593766    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f777f-960b-432b-b5a5-191baaa576de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.695849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.695809    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lhkp\" (UniqueName: \"kubernetes.io/projected/305f777f-960b-432b-b5a5-191baaa576de-kube-api-access-7lhkp\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.696072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.695881    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f777f-960b-432b-b5a5-191baaa576de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.696072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.695961    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f777f-960b-432b-b5a5-191baaa576de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.696685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.696616    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f777f-960b-432b-b5a5-191baaa576de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.698794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.698768    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f777f-960b-432b-b5a5-191baaa576de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.721015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.720987    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lhkp\" (UniqueName: \"kubernetes.io/projected/305f777f-960b-432b-b5a5-191baaa576de-kube-api-access-7lhkp\") pod \"maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:00.775513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:00.775476    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:01.100428 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:25:01.100394    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305f777f_960b_432b_b5a5_191baaa576de.slice/crio-d286f8a884f8d4a1b90b60263571a67209c0f9cb20906a024bc977e8053d4742 WatchSource:0}: Error finding container d286f8a884f8d4a1b90b60263571a67209c0f9cb20906a024bc977e8053d4742: Status 404 returned error can't find the container with id d286f8a884f8d4a1b90b60263571a67209c0f9cb20906a024bc977e8053d4742
Nov 02 15:25:01.100589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:01.100535    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"]
Nov 02 15:25:01.105869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:01.105836    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:01.479688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:01.476625    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d" event={"ID":"305f777f-960b-432b-b5a5-191baaa576de","Type":"ContainerStarted","Data":"d286f8a884f8d4a1b90b60263571a67209c0f9cb20906a024bc977e8053d4742"}
Nov 02 15:25:02.222841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:02.222124    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:02.386807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:02.386759    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:02.486777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:02.486689    7019 generic.go:356] "Generic (PLEG): container finished" podID="305f777f-960b-432b-b5a5-191baaa576de" containerID="183574c56e4a37e06c7b2e2e3c805b3463935b30091d8cb4fcbd8c221064d996" exitCode=0
Nov 02 15:25:02.486777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:02.486728    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d" event={"ID":"305f777f-960b-432b-b5a5-191baaa576de","Type":"ContainerDied","Data":"183574c56e4a37e06c7b2e2e3c805b3463935b30091d8cb4fcbd8c221064d996"}
Nov 02 15:25:03.689715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.689687    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:03.846530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.846449    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305f777f-960b-432b-b5a5-191baaa576de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f777f-960b-432b-b5a5-191baaa576de-maas-api-xks-ca\") pod \"305f777f-960b-432b-b5a5-191baaa576de\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") "
Nov 02 15:25:03.846711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.846573    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305f777f-960b-432b-b5a5-191baaa576de-kube-api-access-7lhkp\" (UniqueName: \"kubernetes.io/projected/305f777f-960b-432b-b5a5-191baaa576de-kube-api-access-7lhkp\") pod \"305f777f-960b-432b-b5a5-191baaa576de\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") "
Nov 02 15:25:03.846711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.846685    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305f777f-960b-432b-b5a5-191baaa576de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f777f-960b-432b-b5a5-191baaa576de-maas-api-openshift-ca\") pod \"305f777f-960b-432b-b5a5-191baaa576de\" (UID: \"305f777f-960b-432b-b5a5-191baaa576de\") "
Nov 02 15:25:03.847052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.847024    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305f777f-960b-432b-b5a5-191baaa576de-maas-api-openshift-ca" pod "305f777f-960b-432b-b5a5-191baaa576de" (UID: "305f777f-960b-432b-b5a5-191baaa576de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:03.847311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.847295    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305f777f-960b-432b-b5a5-191baaa576de-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:03.848518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.848498    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305f777f-960b-432b-b5a5-191baaa576de-maas-api-xks-ca" pod "305f777f-960b-432b-b5a5-191baaa576de" (UID: "305f777f-960b-432b-b5a5-191baaa576de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:25:03.848671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.848642    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305f777f-960b-432b-b5a5-191baaa576de-kube-api-access-7lhkp" pod "305f777f-960b-432b-b5a5-191baaa576de" (UID: "305f777f-960b-432b-b5a5-191baaa576de"). InnerVolumeSpecName "kube-api-access-7lhkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:03.949319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.949275    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lhkp\" (UniqueName: \"kubernetes.io/projected/305f777f-960b-432b-b5a5-191baaa576de-kube-api-access-7lhkp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:03.949319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:03.949319    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305f777f-960b-432b-b5a5-191baaa576de-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.039701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.035347    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:04.510383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.510355    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"
Nov 02 15:25:04.510523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.510379    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d" event={"ID":"305f777f-960b-432b-b5a5-191baaa576de","Type":"ContainerDied","Data":"d286f8a884f8d4a1b90b60263571a67209c0f9cb20906a024bc977e8053d4742"}
Nov 02 15:25:04.510523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.510435    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d286f8a884f8d4a1b90b60263571a67209c0f9cb20906a024bc977e8053d4742"
Nov 02 15:25:04.513519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.512972    7019 generic.go:356] "Generic (PLEG): container finished" podID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerID="141095b0ceff7217a7759f6aad807129ff8310b7ee858f1103560f1e7c19f5c9" exitCode=0
Nov 02 15:25:04.513519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.513155    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" event={"ID":"7914308d-0646-4ff8-a269-63cfe6b0b2bb","Type":"ContainerDied","Data":"141095b0ceff7217a7759f6aad807129ff8310b7ee858f1103560f1e7c19f5c9"}
Nov 02 15:25:04.513519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.513207    7019 scope.go:122] "RemoveContainer" containerID="05cc727b55d2f1e9a25cdfc42deae71e2126c5370db688f13b48c645964625f6"
Nov 02 15:25:04.678597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.678571    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:25:04.863486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863450    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-socket\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.863958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863577    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-token\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-token\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.863958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863627    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-envoy\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.863958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863776    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-credential-socket\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.863958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863777    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-socket" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:04.863958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863921    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-certs\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.863958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863954    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-podinfo\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.864351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.863981    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istiod-ca-cert\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.864351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.864050    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-credential-socket" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:04.864351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.864063    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-kube-api-access-dhqbd\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-kube-api-access-dhqbd\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.864351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.864147    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-data\") pod \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\" (UID: \"7914308d-0646-4ff8-a269-63cfe6b0b2bb\") "
Nov 02 15:25:04.864351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.864204    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-certs" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:04.864641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.864413    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istiod-ca-cert" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:04.864641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.864438    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-data" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:04.865100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.865075    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.865192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.865103    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.865192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.865117    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.865192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.865132    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.865192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.865145    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.866328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.866295    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-token" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:04.866557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.866533    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-kube-api-access-dhqbd" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "kube-api-access-dhqbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:04.866798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.866777    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-envoy" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:25:04.866960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.866942    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-podinfo" pod "7914308d-0646-4ff8-a269-63cfe6b0b2bb" (UID: "7914308d-0646-4ff8-a269-63cfe6b0b2bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:25:04.967181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.967102    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.967181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.967141    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhqbd\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-kube-api-access-dhqbd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.967181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.967156    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:04.967181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:04.967170    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7914308d-0646-4ff8-a269-63cfe6b0b2bb-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:05.523478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.523437    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn" event={"ID":"7914308d-0646-4ff8-a269-63cfe6b0b2bb","Type":"ContainerDied","Data":"68d2459a2643f2a6b9fdfc1bad5a686891ec0e37aaa51fb70595db15203b6cd0"}
Nov 02 15:25:05.523478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.523451    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"
Nov 02 15:25:05.523760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.523507    7019 scope.go:122] "RemoveContainer" containerID="141095b0ceff7217a7759f6aad807129ff8310b7ee858f1103560f1e7c19f5c9"
Nov 02 15:25:05.604154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.604117    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"]
Nov 02 15:25:05.616410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.616382    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1cdbe6-openshift-default-59bc4b69dc-cb4hn"]
Nov 02 15:25:05.653813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.653011    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:05.929674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.922102    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" path="/var/lib/kubelet/pods/7914308d-0646-4ff8-a269-63cfe6b0b2bb/volumes"
Nov 02 15:25:05.965670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:05.965284    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:25:07.317445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:07.317414    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:07.918132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:07.918096    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:08.041485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:08.041446    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"]
Nov 02 15:25:08.053006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:08.052971    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-db75ff-vnd6w"]
Nov 02 15:25:08.089331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:08.089289    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:08.245496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:08.245099    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:09.864320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.864288    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f893f0-b13d-4c4d-8be1-4f750ca074ff" path="/var/lib/kubelet/pods/73f893f0-b13d-4c4d-8be1-4f750ca074ff/volumes"
Nov 02 15:25:09.934972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.934938    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"]
Nov 02 15:25:09.935707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.935679    7019 cpu_manager.go:414] "containerMap: removing container" podUID="305f777f-960b-432b-b5a5-191baaa576de" containerName="revoke-keys"
Nov 02 15:25:09.935707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.935700    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305f777f-960b-432b-b5a5-191baaa576de" containerName="revoke-keys"
Nov 02 15:25:09.935841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.935744    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy"
Nov 02 15:25:09.935841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.935751    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy"
Nov 02 15:25:09.935920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.935902    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy"
Nov 02 15:25:09.936112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.936094    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="305f777f-960b-432b-b5a5-191baaa576de" containerName="revoke-keys"
Nov 02 15:25:09.940643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.940624    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:09.986297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.985032    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2abeaa\"/\"test-model-2abeaa-kserve-self-signed-certs\""
Nov 02 15:25:09.988992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.985108    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2abeaa\"/\"openshift-service-ca.crt\""
Nov 02 15:25:09.988992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.987198    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2abeaa\"/\"kube-root-ca.crt\""
Nov 02 15:25:09.993675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:09.992778    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2abeaa\"/\"default-dockercfg-pqrm2\""
Nov 02 15:25:10.030307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.026214    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"]
Nov 02 15:25:10.030496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.030345    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:10.104697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.102933    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-model-cache\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.104697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.102999    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-dshm\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.104697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.103119    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tmp-dir\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.104697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.103276    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-home\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.104697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.103399    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv25s\" (UniqueName: \"kubernetes.io/projected/816f42c0-98fd-409c-8a8c-52350a1ba4c1-kube-api-access-lv25s\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.104697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.103514    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tls-certs\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.203746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.192607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.206688    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv25s\" (UniqueName: \"kubernetes.io/projected/816f42c0-98fd-409c-8a8c-52350a1ba4c1-kube-api-access-lv25s\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.206802    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tls-certs\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.206842    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-model-cache\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.206878    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-dshm\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.206989    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tmp-dir\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.207111    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-home\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.207740    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-home\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.208993    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tmp-dir\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.210710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.209216    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-model-cache\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.219666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.211924    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-dshm\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.237729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.232307    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tls-certs\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.242777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.238622    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv25s\" (UniqueName: \"kubernetes.io/projected/816f42c0-98fd-409c-8a8c-52350a1ba4c1-kube-api-access-lv25s\") pod \"test-model-2abeaa-kserve-754b74f5f-4dqsq\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") " pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.304696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.301930    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:10.812668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.808093    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"]
Nov 02 15:25:10.812668 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:25:10.808109    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod816f42c0_98fd_409c_8a8c_52350a1ba4c1.slice/crio-42405957718f54c14188b56f79eed585c8b1f9b19a239c3cfec73ed027a90810 WatchSource:0}: Error finding container 42405957718f54c14188b56f79eed585c8b1f9b19a239c3cfec73ed027a90810: Status 404 returned error can't find the container with id 42405957718f54c14188b56f79eed585c8b1f9b19a239c3cfec73ed027a90810
Nov 02 15:25:10.906182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:10.906152    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"]
Nov 02 15:25:11.054667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:11.053265    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:11.218671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:11.214491    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:11.841673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:11.841405    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq" event={"ID":"816f42c0-98fd-409c-8a8c-52350a1ba4c1","Type":"ContainerStarted","Data":"94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14"}
Nov 02 15:25:11.841673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:11.841460    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq" event={"ID":"816f42c0-98fd-409c-8a8c-52350a1ba4c1","Type":"ContainerStarted","Data":"42405957718f54c14188b56f79eed585c8b1f9b19a239c3cfec73ed027a90810"}
Nov 02 15:25:11.906075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:11.906002    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq" podStartSLOduration=2.905983227 podStartE2EDuration="2.905983227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:11.899556052 +0000 UTC m=+1706.153509674" watchObservedRunningTime="2026-11-02 15:25:11.905983227 +0000 UTC m=+1706.159936852"
Nov 02 15:25:12.124229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:12.124192    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:12.303079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:12.303046    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:13.602303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:13.602261    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:13.741198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:13.741158    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:14.150278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:14.150229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:14.943920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:14.943877    7019 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:25:14.944307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:14.943953    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:25:15.315245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:15.315121    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:15.907930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:15.907888    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq" podUID="ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" containerName="payload-pre-processing" containerID="cri-o://fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45" gracePeriod=25
Nov 02 15:25:16.164251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.164158    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"]
Nov 02 15:25:16.165274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.165250    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy"
Nov 02 15:25:16.165388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.165275    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy"
Nov 02 15:25:16.165587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.165565    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7914308d-0646-4ff8-a269-63cfe6b0b2bb" containerName="istio-proxy"
Nov 02 15:25:16.170055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.170032    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.196702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.196668    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"]
Nov 02 15:25:16.199995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.199564    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:25:16.199995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.199722    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:25:16.243699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.243670    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:25:16.302770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.302734    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-plugins-config-volume\") pod \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") "
Nov 02 15:25:16.302963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.302858    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-kube-api-access-b6sjm\" (UniqueName: \"kubernetes.io/projected/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-kube-api-access-b6sjm\") pod \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\" (UID: \"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15\") "
Nov 02 15:25:16.303491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.303451    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.303632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.303601    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-plugins-config-volume" pod "ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" (UID: "ca81c9f4-1de7-4bfa-abfe-7a05d5839b15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:16.303714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.303611    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrdq\" (UniqueName: \"kubernetes.io/projected/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-kube-api-access-6hrdq\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.303797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.303772    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.303980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.303964    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:16.305802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.305775    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-kube-api-access-b6sjm" pod "ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" (UID: "ca81c9f4-1de7-4bfa-abfe-7a05d5839b15"). InnerVolumeSpecName "kube-api-access-b6sjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:16.405309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.405274    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrdq\" (UniqueName: \"kubernetes.io/projected/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-kube-api-access-6hrdq\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.405505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.405354    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.405579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.405560    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.405700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.405681    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6sjm\" (UniqueName: \"kubernetes.io/projected/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15-kube-api-access-b6sjm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:16.406272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.406242    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.408138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.408112    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.430602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.430549    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrdq\" (UniqueName: \"kubernetes.io/projected/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-kube-api-access-6hrdq\") pod \"maas-api-revoke-sub-396b8a5a-vdhds\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.505065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.505022    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:16.833533 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:25:16.833487    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c7cc07_5f41_4eb9_9fa3_facf540edbf1.slice/crio-ba751bcadb1a0919ca155f01f9e0b398595830ef3386701b1e2ca53511403f13 WatchSource:0}: Error finding container ba751bcadb1a0919ca155f01f9e0b398595830ef3386701b1e2ca53511403f13: Status 404 returned error can't find the container with id ba751bcadb1a0919ca155f01f9e0b398595830ef3386701b1e2ca53511403f13
Nov 02 15:25:16.838258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.838226    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"]
Nov 02 15:25:16.888176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.888103    7019 generic.go:356] "Generic (PLEG): container finished" podID="ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" containerID="fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45" exitCode=0
Nov 02 15:25:16.888176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.888160    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"
Nov 02 15:25:16.888496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.888184    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq" event={"ID":"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15","Type":"ContainerDied","Data":"fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45"}
Nov 02 15:25:16.888496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.888243    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq" event={"ID":"ca81c9f4-1de7-4bfa-abfe-7a05d5839b15","Type":"ContainerDied","Data":"d1648ffa518091cfbe0165851f53670fbc99fae41f6e0150fdf5e88694ac0779"}
Nov 02 15:25:16.888496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.888274    7019 scope.go:122] "RemoveContainer" containerID="fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45"
Nov 02 15:25:16.890327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.890299    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds" event={"ID":"28c7cc07-5f41-4eb9-9fa3-facf540edbf1","Type":"ContainerStarted","Data":"ba751bcadb1a0919ca155f01f9e0b398595830ef3386701b1e2ca53511403f13"}
Nov 02 15:25:16.898970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.898939    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:16.900573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.900550    7019 scope.go:122] "RemoveContainer" containerID="fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45"
Nov 02 15:25:16.900952 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:25:16.900905    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45\": container with ID starting with fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45 not found: ID does not exist" containerID="fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45"
Nov 02 15:25:16.901058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.900946    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45"} err="failed to get container status \"fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45\": rpc error: code = NotFound desc = could not find container \"fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45\": container with ID starting with fa193b7dd48ebd575e75bd5f7d34085df6a973dcd608784b5db491e804d75d45 not found: ID does not exist"
Nov 02 15:25:16.964206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.964138    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"]
Nov 02 15:25:16.972768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:16.970508    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ec02d-65b8db9bd7-vc6nq"]
Nov 02 15:25:17.864608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:17.864574    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" path="/var/lib/kubelet/pods/ca81c9f4-1de7-4bfa-abfe-7a05d5839b15/volumes"
Nov 02 15:25:17.902345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:17.902292    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds" event={"ID":"28c7cc07-5f41-4eb9-9fa3-facf540edbf1","Type":"ContainerStarted","Data":"4b36bba05a05151a42389480b533b4b6b2b7405c16a74c092590e98f4b446f29"}
Nov 02 15:25:18.337511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.337472    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:18.639670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.638678    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:18.969574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.968006    7019 generic.go:356] "Generic (PLEG): container finished" podID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerID="bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984" exitCode=0
Nov 02 15:25:18.969574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.968288    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" event={"ID":"92d092a7-8dbc-4d1f-a67f-ad9ca8645796","Type":"ContainerDied","Data":"bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984"}
Nov 02 15:25:18.969574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.969400    7019 scope.go:122] "RemoveContainer" containerID="bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984"
Nov 02 15:25:18.972776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.972556    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:25:18.972776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.972594    7019 generic.go:356] "Generic (PLEG): container finished" podID="28c7cc07-5f41-4eb9-9fa3-facf540edbf1" containerID="4b36bba05a05151a42389480b533b4b6b2b7405c16a74c092590e98f4b446f29" exitCode=0
Nov 02 15:25:18.972776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.972671    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:25:18.972776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.972723    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:25:18.972776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:18.972731    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds" event={"ID":"28c7cc07-5f41-4eb9-9fa3-facf540edbf1","Type":"ContainerDied","Data":"4b36bba05a05151a42389480b533b4b6b2b7405c16a74c092590e98f4b446f29"}
Nov 02 15:25:19.003599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.003559    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:19.046091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.046039    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds" podStartSLOduration=3.046024934 podStartE2EDuration="3.046024934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:17.960861011 +0000 UTC m=+1712.214814635" watchObservedRunningTime="2026-11-02 15:25:19.046024934 +0000 UTC m=+1713.299978558"
Nov 02 15:25:19.123034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.123000    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"]
Nov 02 15:25:19.128830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.128801    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-815746-gn92t"]
Nov 02 15:25:19.176348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.176314    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:19.346963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.346869    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:19.653262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.653211    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:19.867698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:19.866058    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55bce1c-05d1-4992-a0de-60232fe924ad" path="/var/lib/kubelet/pods/e55bce1c-05d1-4992-a0de-60232fe924ad/volumes"
Nov 02 15:25:20.002371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.002274    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" event={"ID":"92d092a7-8dbc-4d1f-a67f-ad9ca8645796","Type":"ContainerStarted","Data":"fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0"}
Nov 02 15:25:20.113723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.113627    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:20.244838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.244813    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:20.251317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.251291    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-kube-api-access-6hrdq\" (UniqueName: \"kubernetes.io/projected/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-kube-api-access-6hrdq\") pod \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") "
Nov 02 15:25:20.251438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.251380    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-openshift-ca\") pod \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") "
Nov 02 15:25:20.251438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.251426    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-xks-ca\") pod \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\" (UID: \"28c7cc07-5f41-4eb9-9fa3-facf540edbf1\") "
Nov 02 15:25:20.251779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.251752    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-openshift-ca" pod "28c7cc07-5f41-4eb9-9fa3-facf540edbf1" (UID: "28c7cc07-5f41-4eb9-9fa3-facf540edbf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:20.252184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.252164    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:20.253531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.253478    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-kube-api-access-6hrdq" pod "28c7cc07-5f41-4eb9-9fa3-facf540edbf1" (UID: "28c7cc07-5f41-4eb9-9fa3-facf540edbf1"). InnerVolumeSpecName "kube-api-access-6hrdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:20.253619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.253558    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-xks-ca" pod "28c7cc07-5f41-4eb9-9fa3-facf540edbf1" (UID: "28c7cc07-5f41-4eb9-9fa3-facf540edbf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:25:20.300175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.299980    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:20.303345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.303309    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:20.318962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.318934    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:20.353558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.353526    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrdq\" (UniqueName: \"kubernetes.io/projected/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-kube-api-access-6hrdq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:20.353760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.353626    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c7cc07-5f41-4eb9-9fa3-facf540edbf1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:20.462809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.462768    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:25:20.463256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.463234    7019 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:20.463319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.463294    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 15:25:20.988668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:20.988343    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:21.030667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:21.029745    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"
Nov 02 15:25:21.037636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:21.037551    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds" event={"ID":"28c7cc07-5f41-4eb9-9fa3-facf540edbf1","Type":"ContainerDied","Data":"ba751bcadb1a0919ca155f01f9e0b398595830ef3386701b1e2ca53511403f13"}
Nov 02 15:25:21.037636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:21.037603    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba751bcadb1a0919ca155f01f9e0b398595830ef3386701b1e2ca53511403f13"
Nov 02 15:25:21.051611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:21.051587    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:25:21.462758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:21.462711    7019 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:21.462958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:21.462788    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 15:25:22.462179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:22.462138    7019 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:22.462568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:22.462203    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 15:25:22.607197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:22.607160    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:22.723186 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:25:22.723099    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:25:23.462788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:23.462745    7019 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:23.463262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:23.462806    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 15:25:24.108795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:24.108734    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:24.462775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:24.462742    7019 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body=
Nov 02 15:25:24.462962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:24.462796    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused"
Nov 02 15:25:25.100389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:25.100346    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:25.253874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:25.253805    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:25.494854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:25.494814    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:25:25.495340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:25.495315    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:25:25.496293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:25.496273    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:25:25.509731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:25.509698    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:25.997296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:25.997262    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:26.159352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.159321    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"]
Nov 02 15:25:26.160059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.160043    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" containerName="payload-pre-processing"
Nov 02 15:25:26.160108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.160061    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" containerName="payload-pre-processing"
Nov 02 15:25:26.160108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.160075    7019 cpu_manager.go:414] "containerMap: removing container" podUID="28c7cc07-5f41-4eb9-9fa3-facf540edbf1" containerName="revoke-keys"
Nov 02 15:25:26.160108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.160081    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c7cc07-5f41-4eb9-9fa3-facf540edbf1" containerName="revoke-keys"
Nov 02 15:25:26.160210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.160198    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca81c9f4-1de7-4bfa-abfe-7a05d5839b15" containerName="payload-pre-processing"
Nov 02 15:25:26.160210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.160209    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c7cc07-5f41-4eb9-9fa3-facf540edbf1" containerName="revoke-keys"
Nov 02 15:25:26.163739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.163718    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.184715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.184681    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"]
Nov 02 15:25:26.224634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.224585    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.225007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.224988    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsszp\" (UniqueName: \"kubernetes.io/projected/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-kube-api-access-lsszp\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.225290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.225125    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.225823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.225803    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:25:26.225943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.225925    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:25:26.327149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.327051    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.327316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.327233    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.327444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.327427    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsszp\" (UniqueName: \"kubernetes.io/projected/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-kube-api-access-lsszp\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.327939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.327912    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.329454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.329434    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.352995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.352962    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsszp\" (UniqueName: \"kubernetes.io/projected/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-kube-api-access-lsszp\") pod \"maas-api-revoke-keys-e2e-api-b-21ece2-2f87g\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.479277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.479237    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:26.788845 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:25:26.788814    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e0053d_0767_4dda_a96b_2d7ffcc4c39b.slice/crio-6b3e75ff00f966dffa7cbea8a0b9a54b63385b8bb57e193bf00d0ca4bc514334 WatchSource:0}: Error finding container 6b3e75ff00f966dffa7cbea8a0b9a54b63385b8bb57e193bf00d0ca4bc514334: Status 404 returned error can't find the container with id 6b3e75ff00f966dffa7cbea8a0b9a54b63385b8bb57e193bf00d0ca4bc514334
Nov 02 15:25:26.790011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:26.789980    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"]
Nov 02 15:25:27.095399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:27.095303    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g" event={"ID":"65e0053d-0767-4dda-a96b-2d7ffcc4c39b","Type":"ContainerStarted","Data":"6b3e75ff00f966dffa7cbea8a0b9a54b63385b8bb57e193bf00d0ca4bc514334"}
Nov 02 15:25:28.106045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:28.083191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:28.248055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:28.248010    7019 generic.go:356] "Generic (PLEG): container finished" podID="65e0053d-0767-4dda-a96b-2d7ffcc4c39b" containerID="33735ea992cd905b3c3c58ffef036121cf0cbfe92ba12ab9371df29701ccf4fc" exitCode=0
Nov 02 15:25:28.251946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:28.251906    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g" event={"ID":"65e0053d-0767-4dda-a96b-2d7ffcc4c39b","Type":"ContainerDied","Data":"33735ea992cd905b3c3c58ffef036121cf0cbfe92ba12ab9371df29701ccf4fc"}
Nov 02 15:25:28.381578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:28.381544    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:28.546418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:28.546381    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:28.925827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:28.925793    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:29.271238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.271146    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:29.535641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.535558    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:29.728507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.728465    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-kube-api-access-lsszp\" (UniqueName: \"kubernetes.io/projected/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-kube-api-access-lsszp\") pod \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") "
Nov 02 15:25:29.729013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.728983    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-openshift-ca\") pod \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") "
Nov 02 15:25:29.729405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.729377    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-xks-ca\") pod \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\" (UID: \"65e0053d-0767-4dda-a96b-2d7ffcc4c39b\") "
Nov 02 15:25:29.729972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.729946    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-openshift-ca" pod "65e0053d-0767-4dda-a96b-2d7ffcc4c39b" (UID: "65e0053d-0767-4dda-a96b-2d7ffcc4c39b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:29.733309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.733139    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-xks-ca" pod "65e0053d-0767-4dda-a96b-2d7ffcc4c39b" (UID: "65e0053d-0767-4dda-a96b-2d7ffcc4c39b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:25:29.750002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.747238    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-kube-api-access-lsszp" pod "65e0053d-0767-4dda-a96b-2d7ffcc4c39b" (UID: "65e0053d-0767-4dda-a96b-2d7ffcc4c39b"). InnerVolumeSpecName "kube-api-access-lsszp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:29.831827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.831724    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:29.831827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.831760    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsszp\" (UniqueName: \"kubernetes.io/projected/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-kube-api-access-lsszp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:29.831827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.831770    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e0053d-0767-4dda-a96b-2d7ffcc4c39b-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:29.853467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:29.853334    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:30.304248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:30.304207    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g" event={"ID":"65e0053d-0767-4dda-a96b-2d7ffcc4c39b","Type":"ContainerDied","Data":"6b3e75ff00f966dffa7cbea8a0b9a54b63385b8bb57e193bf00d0ca4bc514334"}
Nov 02 15:25:30.304793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:30.304269    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b3e75ff00f966dffa7cbea8a0b9a54b63385b8bb57e193bf00d0ca4bc514334"
Nov 02 15:25:30.304793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:30.304349    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"
Nov 02 15:25:31.592531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:31.592505    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:32.721624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:32.721589    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:33.056045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:33.055944    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:33.465723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:33.465685    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:34.412393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:34.412281    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:34.589697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:34.589556    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:35.369238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:35.369201    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:36.457094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:36.457052    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"]
Nov 02 15:25:36.644874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:36.644838    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:36.784068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:36.783958    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:37.211274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:37.211238    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:37.746198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:37.746161    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:38.961288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:38.961250    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:40.630024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:40.629991    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:41.458469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.458420    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq" podUID="2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" containerName="payload-processing" containerID="cri-o://76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5" gracePeriod=25
Nov 02 15:25:41.689889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.689535    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:41.799573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.799534    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:25:41.920604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.920566    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-kube-api-access-k2jtm\" (UniqueName: \"kubernetes.io/projected/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-kube-api-access-k2jtm\") pod \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") "
Nov 02 15:25:41.920824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.920698    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-plugins-config-volume\") pod \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\" (UID: \"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9\") "
Nov 02 15:25:41.921135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.921103    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-plugins-config-volume" pod "2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" (UID: "2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:25:41.923756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.923707    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-kube-api-access-k2jtm" pod "2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" (UID: "2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9"). InnerVolumeSpecName "kube-api-access-k2jtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:25:41.924117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.924092    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2jtm\" (UniqueName: \"kubernetes.io/projected/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-kube-api-access-k2jtm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:41.924225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.924121    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:25:41.969979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:41.969931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:42.430551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.430519    7019 generic.go:356] "Generic (PLEG): container finished" podID="2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" containerID="76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5" exitCode=0
Nov 02 15:25:42.430775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.430575    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"
Nov 02 15:25:42.430775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.430603    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq" event={"ID":"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9","Type":"ContainerDied","Data":"76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5"}
Nov 02 15:25:42.430775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.430689    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq" event={"ID":"2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9","Type":"ContainerDied","Data":"5ccdd9362250d40d1cfb824c452d6e1045284c160327de92118c5a70935fc1f2"}
Nov 02 15:25:42.430775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.430712    7019 scope.go:122] "RemoveContainer" containerID="76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5"
Nov 02 15:25:42.444492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.444473    7019 scope.go:122] "RemoveContainer" containerID="76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5"
Nov 02 15:25:42.444825 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:25:42.444805    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5\": container with ID starting with 76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5 not found: ID does not exist" containerID="76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5"
Nov 02 15:25:42.444920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.444839    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5"} err="failed to get container status \"76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5\": rpc error: code = NotFound desc = could not find container \"76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5\": container with ID starting with 76b9c08c1a719dce72f8251a426bcca7332d1cd04c30a20286d91c73c09b7ff5 not found: ID does not exist"
Nov 02 15:25:42.507670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.500592    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"]
Nov 02 15:25:42.533460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.532512    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-21ece2-9c8b756f-qrlbq"]
Nov 02 15:25:42.537569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:42.537528    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:43.886098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:43.884732    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" path="/var/lib/kubelet/pods/2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9/volumes"
Nov 02 15:25:44.658572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:44.658522    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:44.856385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:44.856287    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:46.543816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:46.543777    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:47.300996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:47.300950    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:47.815189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:47.815161    7019 scope.go:122] "RemoveContainer" containerID="3712e1581924667be4c0442a609c8bbb044a15eeecbd936effa5b0b576f39c98"
Nov 02 15:25:47.825947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:47.825927    7019 scope.go:122] "RemoveContainer" containerID="291c234197905045a208f6b0a2b2ac9d7f27e7376df32a76bfa14c789b99458a"
Nov 02 15:25:47.836052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:47.836030    7019 scope.go:122] "RemoveContainer" containerID="950aad49c6f509dd7ac0d5e05f8e4208d0ccbc68bff1c93fb82ab0c6b650ae42"
Nov 02 15:25:48.262413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:48.262304    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:49.894910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:49.894863    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:51.494960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:51.494919    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:52.708402 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:25:52.708368    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:25:53.065241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:53.065139    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:54.640378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:54.640163    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:56.193108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:56.193071    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:57.813569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:57.813531    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:25:59.396801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:25:59.396763    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:01.067774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:01.067732    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:01.216696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:01.216630    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:02.705633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:02.705582    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:04.416575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:04.408584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:05.998258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:05.998220    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:07.701061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:07.701014    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:09.373951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:09.373910    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:09.523307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:09.523257    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:11.161566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:11.161531    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:12.994668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:12.988385    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:13.903242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.903209    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"]
Nov 02 15:26:13.904080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.904055    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" containerName="payload-processing"
Nov 02 15:26:13.904080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.904077    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" containerName="payload-processing"
Nov 02 15:26:13.904283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.904113    7019 cpu_manager.go:414] "containerMap: removing container" podUID="65e0053d-0767-4dda-a96b-2d7ffcc4c39b" containerName="revoke-keys"
Nov 02 15:26:13.904283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.904120    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e0053d-0767-4dda-a96b-2d7ffcc4c39b" containerName="revoke-keys"
Nov 02 15:26:13.904396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.904314    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e0053d-0767-4dda-a96b-2d7ffcc4c39b" containerName="revoke-keys"
Nov 02 15:26:13.904396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.904343    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2309c81c-8ca6-4ebf-beef-eff4bd1ad5d9" containerName="payload-processing"
Nov 02 15:26:13.908521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.908500    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:13.924899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.924880    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a03e84-openshift-default-dockercfg-fqmh5\""
Nov 02 15:26:13.942542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:13.942507    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"]
Nov 02 15:26:14.055165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055128    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-token\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055254    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-envoy\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055325    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-credential-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055477    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13b6c4-d185-42a0-bc81-0a8461455755-istiod-ca-cert\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055529    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-certs\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055599    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-data\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055683    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvpf\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-kube-api-access-jhvpf\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055704    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-podinfo\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.055793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.055759    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.156977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.156880    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-envoy\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.156977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.156968    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-credential-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157063    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13b6c4-d185-42a0-bc81-0a8461455755-istiod-ca-cert\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157123    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-certs\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157167    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-data\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157235    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvpf\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-kube-api-access-jhvpf\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157369    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-podinfo\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157439    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-credential-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157441    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157491    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-certs\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157540    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-token\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157585    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-data\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157818    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-socket\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.157893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.157834    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13b6c4-d185-42a0-bc81-0a8461455755-istiod-ca-cert\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.159346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.159328    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-envoy\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.160053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.160029    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-podinfo\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.185237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.185199    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-token\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.185386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.185298    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvpf\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-kube-api-access-jhvpf\") pod \"e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") " pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.223737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.223708    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:14.548535 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:26:14.548504    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b13b6c4_d185_42a0_bc81_0a8461455755.slice/crio-188a2ff0457fdb69e3dabb0e032606a78df9582790b6084181b53bc0355b3c51 WatchSource:0}: Error finding container 188a2ff0457fdb69e3dabb0e032606a78df9582790b6084181b53bc0355b3c51: Status 404 returned error can't find the container with id 188a2ff0457fdb69e3dabb0e032606a78df9582790b6084181b53bc0355b3c51
Nov 02 15:26:14.551522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.551484    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:26:14.551613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.551555    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:26:14.551613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.551583    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:26:14.553292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.553269    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"]
Nov 02 15:26:14.682079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.682014    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:14.729717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.726692    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" event={"ID":"9b13b6c4-d185-42a0-bc81-0a8461455755","Type":"ContainerStarted","Data":"c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096"}
Nov 02 15:26:14.729717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.726751    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" event={"ID":"9b13b6c4-d185-42a0-bc81-0a8461455755","Type":"ContainerStarted","Data":"188a2ff0457fdb69e3dabb0e032606a78df9582790b6084181b53bc0355b3c51"}
Nov 02 15:26:14.784411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:14.784347    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podStartSLOduration=1.78432768 podStartE2EDuration="1.78432768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:14.781554744 +0000 UTC m=+1769.035508371" watchObservedRunningTime="2026-11-02 15:26:14.78432768 +0000 UTC m=+1769.038281304"
Nov 02 15:26:15.224286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:15.224251    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:15.225912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:15.225884    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 15:26:15.226051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:15.225933    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 15:26:16.136681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:16.136598    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:16.224205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:16.224160    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 15:26:16.224414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:16.224225    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 15:26:16.833674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:16.828407    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:16.927769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:16.927733    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"]
Nov 02 15:26:16.933402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:16.933371    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ec02d-f2n4d"]
Nov 02 15:26:16.995211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:16.995173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:17.143967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:17.143932    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:17.225137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:17.225088    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 15:26:17.225619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:17.225148    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 15:26:17.528744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:17.528639    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:17.873674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:17.865990    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305f777f-960b-432b-b5a5-191baaa576de" path="/var/lib/kubelet/pods/305f777f-960b-432b-b5a5-191baaa576de/volumes"
Nov 02 15:26:18.224491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:18.224408    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 15:26:18.224491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:18.224466    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 15:26:18.729671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:18.724454    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:19.224384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:19.224349    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 15:26:19.224605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:19.224412    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 15:26:20.224776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:20.224740    7019 patch_prober.go:29] interesting pod/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 02 15:26:20.225208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:20.224796    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 02 15:26:20.339865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:20.339826    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:20.499374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:20.499236    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:20.640430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:20.640397    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:21.640387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:21.640351    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:21.641908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:21.641877    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:21.642810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:21.642780    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:26:22.279697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:22.279590    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:22.714963 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:26:22.714925    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:26:23.993348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:23.993313    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:24.643975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:24.643939    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:24.801871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:24.801836    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:25.040304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:25.040214    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:25.684522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:25.684487    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:26.466270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:26.466230    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:26.999661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:26.999605    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:27.183079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:27.183046    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:27.620259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:27.620217    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:29.395424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:29.395376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:31.114906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:31.114869    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:31.619981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:31.619947    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:31.872863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:31.872766    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:33.152351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.152315    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:33.317334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.317295    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:33.516780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.497834    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:33.658606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.658570    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"]
Nov 02 15:26:33.747711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.747636    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"]
Nov 02 15:26:33.747912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.747799    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:33.779428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.779292    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-80ec03-dockercfg-jzh4r\""
Nov 02 15:26:33.779428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.779358    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-80ec03\""
Nov 02 15:26:33.936848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.936804    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:33.937106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:33.937077    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5z7s\" (UniqueName: \"kubernetes.io/projected/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-kube-api-access-p5z7s\") pod \"payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:34.038594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.038507    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:34.038790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.038680    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5z7s\" (UniqueName: \"kubernetes.io/projected/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-kube-api-access-p5z7s\") pod \"payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:34.039299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.039271    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:34.063978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.063953    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5z7s\" (UniqueName: \"kubernetes.io/projected/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-kube-api-access-p5z7s\") pod \"payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:34.079549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.079517    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:34.393389 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:26:34.393347    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a1ff58_cd58_4663_aa5c_6fbf0d7fcb64.slice/crio-3c5e759d5c4d012114709780bb4659e6063ee68e316e4a28fd0c29e67491ffb4 WatchSource:0}: Error finding container 3c5e759d5c4d012114709780bb4659e6063ee68e316e4a28fd0c29e67491ffb4: Status 404 returned error can't find the container with id 3c5e759d5c4d012114709780bb4659e6063ee68e316e4a28fd0c29e67491ffb4
Nov 02 15:26:34.394287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.394259    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"]
Nov 02 15:26:34.839490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.839398    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:34.959373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.959335    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" event={"ID":"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64","Type":"ContainerStarted","Data":"f0a120c25e68f981657826ecf66408f122918eb58cd338b1a83d282de884edcb"}
Nov 02 15:26:34.959579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.959436    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" event={"ID":"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64","Type":"ContainerStarted","Data":"3c5e759d5c4d012114709780bb4659e6063ee68e316e4a28fd0c29e67491ffb4"}
Nov 02 15:26:34.959579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:34.959475    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:35.219317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:35.219244    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" podStartSLOduration=2.219220873 podStartE2EDuration="2.219220873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:35.004233679 +0000 UTC m=+1789.258187306" watchObservedRunningTime="2026-11-02 15:26:35.219220873 +0000 UTC m=+1789.473174498"
Nov 02 15:26:35.220194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:35.220163    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:26:36.400989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:36.400951    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"]
Nov 02 15:26:36.419672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:36.414009    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5" podUID="0e950221-5e22-4d5f-86b3-ed28cf3e1e73" containerName="maas-api" containerID="cri-o://8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee" gracePeriod=30
Nov 02 15:26:36.565969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:36.565883    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:36.979041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:36.979009    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:26:37.061002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.060969    7019 generic.go:356] "Generic (PLEG): container finished" podID="0e950221-5e22-4d5f-86b3-ed28cf3e1e73" containerID="8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee" exitCode=0
Nov 02 15:26:37.061690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.061222    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5" event={"ID":"0e950221-5e22-4d5f-86b3-ed28cf3e1e73","Type":"ContainerDied","Data":"8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee"}
Nov 02 15:26:37.061690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.061276    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5" event={"ID":"0e950221-5e22-4d5f-86b3-ed28cf3e1e73","Type":"ContainerDied","Data":"bbe9e0086be53634e45bab82c37b3bec74c41c6eb2be64ef34909a1e2762a1fc"}
Nov 02 15:26:37.061690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.061304    7019 scope.go:122] "RemoveContainer" containerID="8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee"
Nov 02 15:26:37.061690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.061461    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"
Nov 02 15:26:37.089227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.089145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:37.115677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.115638    7019 scope.go:122] "RemoveContainer" containerID="8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee"
Nov 02 15:26:37.125958 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:26:37.125918    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee\": container with ID starting with 8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee not found: ID does not exist" containerID="8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee"
Nov 02 15:26:37.126204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.126172    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee"} err="failed to get container status \"8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee\": rpc error: code = NotFound desc = could not find container \"8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee\": container with ID starting with 8e4fbc616fe8a0bf1e119a7062a159eda22c24f4fabec916ae5c5ceb1ed832ee not found: ID does not exist"
Nov 02 15:26:37.137246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.137182    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-maas-api-tls\") pod \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") "
Nov 02 15:26:37.137572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.137548    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-kube-api-access-c7vfj\" (UniqueName: \"kubernetes.io/projected/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-kube-api-access-c7vfj\") pod \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") "
Nov 02 15:26:37.137791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.137772    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-metrics-certs\") pod \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") "
Nov 02 15:26:37.137945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.137925    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-openshift-service-ca\") pod \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\" (UID: \"0e950221-5e22-4d5f-86b3-ed28cf3e1e73\") "
Nov 02 15:26:37.139232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.139209    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-openshift-service-ca" pod "0e950221-5e22-4d5f-86b3-ed28cf3e1e73" (UID: "0e950221-5e22-4d5f-86b3-ed28cf3e1e73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:26:37.150233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.150176    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-metrics-certs" pod "0e950221-5e22-4d5f-86b3-ed28cf3e1e73" (UID: "0e950221-5e22-4d5f-86b3-ed28cf3e1e73"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:26:37.156797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.156739    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-kube-api-access-c7vfj" pod "0e950221-5e22-4d5f-86b3-ed28cf3e1e73" (UID: "0e950221-5e22-4d5f-86b3-ed28cf3e1e73"). InnerVolumeSpecName "kube-api-access-c7vfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:37.160129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.160077    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-maas-api-tls" pod "0e950221-5e22-4d5f-86b3-ed28cf3e1e73" (UID: "0e950221-5e22-4d5f-86b3-ed28cf3e1e73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:26:37.247675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.247612    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:37.248314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.248289    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vfj\" (UniqueName: \"kubernetes.io/projected/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-kube-api-access-c7vfj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:37.248522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.248482    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:37.248522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.248503    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e950221-5e22-4d5f-86b3-ed28cf3e1e73-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:37.317766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.295572    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:37.468224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.468185    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"]
Nov 02 15:26:37.472594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.472558    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-21ece2-2f87g"]
Nov 02 15:26:37.474703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.474521    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:37.912673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.908860    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e0053d-0767-4dda-a96b-2d7ffcc4c39b" path="/var/lib/kubelet/pods/65e0053d-0767-4dda-a96b-2d7ffcc4c39b/volumes"
Nov 02 15:26:37.912673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.909365    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"]
Nov 02 15:26:37.912673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.909411    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-21ece2-6954485c6-prhg5"]
Nov 02 15:26:37.983674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:37.973812    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:38.349728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:38.349485    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:26:39.651691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:39.651233    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:39.865641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:39.865602    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e950221-5e22-4d5f-86b3-ed28cf3e1e73" path="/var/lib/kubelet/pods/0e950221-5e22-4d5f-86b3-ed28cf3e1e73/volumes"
Nov 02 15:26:41.414846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:41.414809    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:43.090971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:43.090938    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:44.755338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:44.754777    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:44.938026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:44.937986    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:45.206888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:45.206850    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"]
Nov 02 15:26:45.207199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:45.207170    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" containerID="cri-o://1b8d7225bc4d471bf4c7f2f8996817a94a007396fea004250806bddfee60f512" gracePeriod=30
Nov 02 15:26:45.324798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:45.323087    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:45.972376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:45.972346    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:26:46.141262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.141212    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:46.461994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.461959    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:46.560449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.560423    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"]
Nov 02 15:26:46.561588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.561556    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0e950221-5e22-4d5f-86b3-ed28cf3e1e73" containerName="maas-api"
Nov 02 15:26:46.561588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.561586    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e950221-5e22-4d5f-86b3-ed28cf3e1e73" containerName="maas-api"
Nov 02 15:26:46.561850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.561758    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e950221-5e22-4d5f-86b3-ed28cf3e1e73" containerName="maas-api"
Nov 02 15:26:46.568087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.568064    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.595038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.595006    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"]
Nov 02 15:26:46.602722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.602331    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-69ff88-openshift-default-dockercfg-xb5gm\""
Nov 02 15:26:46.626315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626278    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.626800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626391    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-data\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.626800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626555    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-certs\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.626800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626618    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-token\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.626800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626701    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvgr\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-kube-api-access-5rvgr\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.626800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626767    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-podinfo\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.626800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626801    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-credential-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.627489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626860    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istiod-ca-cert\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.627489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.626885    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-envoy\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.668555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.665826    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:46.732947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.731796    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istiod-ca-cert\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.733966    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-envoy\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.734154    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.734284    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-data\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.734422    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-certs\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.734485    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-token\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.734575    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvgr\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-kube-api-access-5rvgr\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.734671    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-podinfo\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.734707    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-credential-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.735428    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-certs\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.732623    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istiod-ca-cert\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.735946    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.736669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.736486    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-data\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.746719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.737750    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-credential-socket\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.770678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.764035    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-envoy\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.770678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.764472    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-podinfo\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.812748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.806393    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvgr\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-kube-api-access-5rvgr\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.823685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.813022    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-token\") pod \"e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") " pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.899673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.899554    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:46.975368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:46.975334    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:47.379515 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:26:47.379477    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8db75dc_11e7_4eeb_9c6f_f3128863a73a.slice/crio-2a54a5ab525511722c62e72589ae15bb76764c99a418292c3f066795cf07b291 WatchSource:0}: Error finding container 2a54a5ab525511722c62e72589ae15bb76764c99a418292c3f066795cf07b291: Status 404 returned error can't find the container with id 2a54a5ab525511722c62e72589ae15bb76764c99a418292c3f066795cf07b291
Nov 02 15:26:47.380784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.380757    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"]
Nov 02 15:26:47.382624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.382585    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:26:47.382743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.382671    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:26:47.382743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.382701    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:26:47.426789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.425835    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" event={"ID":"d8db75dc-11e7-4eeb-9c6f-f3128863a73a","Type":"ContainerStarted","Data":"2a54a5ab525511722c62e72589ae15bb76764c99a418292c3f066795cf07b291"}
Nov 02 15:26:47.508114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.508059    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:47.896349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.896311    7019 scope.go:122] "RemoveContainer" containerID="e988464b0a22bfad15015a20508d856615f8a2dc5a7e44e87e6fa3585912efe3"
Nov 02 15:26:47.909660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:47.909619    7019 scope.go:122] "RemoveContainer" containerID="0fdbc1c72910226dc378394705068e30d6ec9132b7c2b9d1b3b8e58bf0e9fa02"
Nov 02 15:26:48.204785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:48.198176    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:48.436157 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:48.436120    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" event={"ID":"d8db75dc-11e7-4eeb-9c6f-f3128863a73a","Type":"ContainerStarted","Data":"7faaeeed3a4c74c7a409a2e0362957607fc832aa300fee7a914f912bfd78793f"}
Nov 02 15:26:48.504137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:48.504006    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podStartSLOduration=2.503989037 podStartE2EDuration="2.503989037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:48.501424538 +0000 UTC m=+1802.755378165" watchObservedRunningTime="2026-11-02 15:26:48.503989037 +0000 UTC m=+1802.757942662"
Nov 02 15:26:48.901125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:48.901080    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:48.909952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:48.909916    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:49.443946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:49.443910    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:49.445081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:49.445063    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:26:49.898172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:49.898133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:50.291953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.291918    7019 patch_prober.go:29] interesting pod/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 15:26:50.292165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.291990    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 15:26:50.455349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.455308    7019 generic.go:356] "Generic (PLEG): container finished" podID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerID="1b8d7225bc4d471bf4c7f2f8996817a94a007396fea004250806bddfee60f512" exitCode=0
Nov 02 15:26:50.457050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.456615    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" event={"ID":"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b","Type":"ContainerDied","Data":"1b8d7225bc4d471bf4c7f2f8996817a94a007396fea004250806bddfee60f512"}
Nov 02 15:26:50.457050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.456702    7019 scope.go:122] "RemoveContainer" containerID="2fc8126cfd2e9eda127011658fac014bb0dbc2df05affd87f389775b2c79eb51"
Nov 02 15:26:50.527829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.527800    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:26:50.620389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620347    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-podinfo\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.620611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620449    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istiod-ca-cert\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.620611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620479    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-socket\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.620611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620504    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-credential-socket\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.620879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620674    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-certs\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.620879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620723    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-data\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.620879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620772    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-kube-api-access-4ggfs\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-kube-api-access-4ggfs\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.621046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620911    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-credential-socket" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:50.621046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620954    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-socket" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:50.621046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.620983    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-token\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-token\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.621209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621042    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-certs" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:50.621209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621062    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-envoy\") pod \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\" (UID: \"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b\") "
Nov 02 15:26:50.621209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621052    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istiod-ca-cert" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:26:50.621209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621140    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-data" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:50.621912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621883    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.622026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621915    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.622026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621932    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.622026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621945    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.622026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.621959    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.623567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.623541    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-envoy" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:26:50.623700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.623672    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-kube-api-access-4ggfs" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "kube-api-access-4ggfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:50.624160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.624134    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-podinfo" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:26:50.625376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.625350    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-token" pod "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" (UID: "b97ad2ce-9fe3-4031-a03f-8f84c1ce443b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:26:50.723367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.723318    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggfs\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-kube-api-access-4ggfs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.723367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.723366    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.723367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.723378    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:50.723636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:50.723397    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:26:51.466138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:51.466088    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf" event={"ID":"b97ad2ce-9fe3-4031-a03f-8f84c1ce443b","Type":"ContainerDied","Data":"34afdd65060199dee4148befda1afd9e51c82673c17e47b1849304ba9c61c4b8"}
Nov 02 15:26:51.466138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:51.466122    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"
Nov 02 15:26:51.466789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:51.466207    7019 scope.go:122] "RemoveContainer" containerID="1b8d7225bc4d471bf4c7f2f8996817a94a007396fea004250806bddfee60f512"
Nov 02 15:26:51.528773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:51.528731    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"]
Nov 02 15:26:51.542215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:51.542180    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-21ece2-openshift-default-548fc68679-48gcf"]
Nov 02 15:26:51.666415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:51.666369    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:51.864223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:51.864193    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" path="/var/lib/kubelet/pods/b97ad2ce-9fe3-4031-a03f-8f84c1ce443b/volumes"
Nov 02 15:26:52.160477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:52.160381    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:52.309333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:52.309293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:52.707315 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:26:52.707277    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:26:53.490564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:53.490519    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:53.896989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:53.896950    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:54.065304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.065261    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:54.181754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.175706    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"]
Nov 02 15:26:54.221731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.213140    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy"
Nov 02 15:26:54.221731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.213178    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy"
Nov 02 15:26:54.221731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.213441    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy"
Nov 02 15:26:54.221731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.213457    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy"
Nov 02 15:26:54.374738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.373743    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:54.404734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.403496    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-69ff88\""
Nov 02 15:26:54.417739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.411120    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"]
Nov 02 15:26:54.452064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.452036    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-69ff88-dockercfg-kts45\""
Nov 02 15:26:54.560188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.560149    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66gf7\" (UniqueName: \"kubernetes.io/projected/2c45682a-e9c5-43f0-8fe0-002daced8c0e-kube-api-access-66gf7\") pod \"payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:54.560414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.560393    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c45682a-e9c5-43f0-8fe0-002daced8c0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:54.662166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.662123    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c45682a-e9c5-43f0-8fe0-002daced8c0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:54.662371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.662304    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66gf7\" (UniqueName: \"kubernetes.io/projected/2c45682a-e9c5-43f0-8fe0-002daced8c0e-kube-api-access-66gf7\") pod \"payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:54.662911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.662890    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c45682a-e9c5-43f0-8fe0-002daced8c0e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:54.687527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.687495    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66gf7\" (UniqueName: \"kubernetes.io/projected/2c45682a-e9c5-43f0-8fe0-002daced8c0e-kube-api-access-66gf7\") pod \"payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:54.751990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:54.751906    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:55.145677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:55.136138    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"]
Nov 02 15:26:55.410771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:55.407356    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:55.508865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:55.508828    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" event={"ID":"2c45682a-e9c5-43f0-8fe0-002daced8c0e","Type":"ContainerStarted","Data":"1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1"}
Nov 02 15:26:55.509062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:55.508892    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" event={"ID":"2c45682a-e9c5-43f0-8fe0-002daced8c0e","Type":"ContainerStarted","Data":"9c52254d7f3a9fcebb1ca00f491b1100073c16b1d96b763ab4c5eecd61ebd6da"}
Nov 02 15:26:55.509062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:55.508914    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:26:55.557057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:55.556852    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" podStartSLOduration=1.556831627 podStartE2EDuration="1.556831627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:55.556410974 +0000 UTC m=+1809.810364601" watchObservedRunningTime="2026-11-02 15:26:55.556831627 +0000 UTC m=+1809.810785251"
Nov 02 15:26:57.045230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:57.045143    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"]
Nov 02 15:26:57.045686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:57.045389    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" containerID="cri-o://ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d" gracePeriod=30
Nov 02 15:26:57.184200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:57.184159    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:26:58.774407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:26:58.774370    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:00.273444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:00.273407    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:00.420301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:00.420265    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:00.788707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:00.788659    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:01.230880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.230845    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:01.422538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.422496    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"]
Nov 02 15:27:01.424013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.423982    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy"
Nov 02 15:27:01.424013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.424012    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97ad2ce-9fe3-4031-a03f-8f84c1ce443b" containerName="istio-proxy"
Nov 02 15:27:01.429139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.429113    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.461055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.461024    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d35a86-openshift-default-dockercfg-9thq7\""
Nov 02 15:27:01.483164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.483060    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"]
Nov 02 15:27:01.503091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503056    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee941490-62a1-40fc-92f9-403df334d9bf-istiod-ca-cert\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503112    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-istio-token\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503239    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-credential-socket\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503268    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-certs\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503310    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee941490-62a1-40fc-92f9-403df334d9bf-istio-podinfo\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503426    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-data\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503460    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqnf\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-kube-api-access-cfqnf\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503553    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-socket\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.503610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.503598    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-envoy\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.610992    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-data\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611057    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqnf\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-kube-api-access-cfqnf\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611097    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-socket\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611147    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-envoy\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611298    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee941490-62a1-40fc-92f9-403df334d9bf-istiod-ca-cert\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611352    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-istio-token\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611417    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-credential-socket\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611442    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-certs\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.611668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611494    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee941490-62a1-40fc-92f9-403df334d9bf-istio-podinfo\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.612347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.611492    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-data\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.612347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.612232    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-credential-socket\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.618677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.612487    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-certs\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.618677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.615791    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-socket\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.651674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.641357    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee941490-62a1-40fc-92f9-403df334d9bf-istiod-ca-cert\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.651674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.651206    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:01.673124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.673056    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee941490-62a1-40fc-92f9-403df334d9bf-istio-podinfo\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.675406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.675335    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-envoy\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.683191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.676206    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqnf\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-kube-api-access-cfqnf\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.684864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.684809    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-istio-token\") pod \"e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") " pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.748165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.748074    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:01.822695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:01.818425    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:02.157931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.157894    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 15:27:02.158113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.157961    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 15:27:02.248163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.248113    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"]
Nov 02 15:27:02.255492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.255435    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:27:02.255615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.255571    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:27:02.255699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.255624    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:27:02.307673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.298685    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:02.410891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.410864    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:27:02.453365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453330    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-kube-api-access-ft5kk\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-kube-api-access-ft5kk\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453467    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-envoy\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453493    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-data\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453549    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6348ed9-1285-4509-8a25-516358d3d83e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6348ed9-1285-4509-8a25-516358d3d83e-istio-podinfo\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453584    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-istio-token\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-istio-token\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453628    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6348ed9-1285-4509-8a25-516358d3d83e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6348ed9-1285-4509-8a25-516358d3d83e-istiod-ca-cert\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453702    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-socket\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453736    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-certs\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.453891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.453773    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-credential-socket\") pod \"e6348ed9-1285-4509-8a25-516358d3d83e\" (UID: \"e6348ed9-1285-4509-8a25-516358d3d83e\") "
Nov 02 15:27:02.455057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.454980    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-credential-socket" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:02.455188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.455090    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-data" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:02.455325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.455302    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-socket" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:02.455465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.455441    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-certs" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:02.455557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.455477    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6348ed9-1285-4509-8a25-516358d3d83e-istiod-ca-cert" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:02.457424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.457386    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-istio-token" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:02.458113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.458073    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-envoy" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:02.458113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.458095    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-kube-api-access-ft5kk" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "kube-api-access-ft5kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:02.458344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.458312    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6348ed9-1285-4509-8a25-516358d3d83e-istio-podinfo" pod "e6348ed9-1285-4509-8a25-516358d3d83e" (UID: "e6348ed9-1285-4509-8a25-516358d3d83e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:27:02.555991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.555950    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6348ed9-1285-4509-8a25-516358d3d83e-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.555991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.555984    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.555991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.555994    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6348ed9-1285-4509-8a25-516358d3d83e-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.555991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.556002    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.556280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.556010    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.556280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.556018    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.556280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.556029    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5kk\" (UniqueName: \"kubernetes.io/projected/e6348ed9-1285-4509-8a25-516358d3d83e-kube-api-access-ft5kk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.556280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.556040    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.556280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.556048    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6348ed9-1285-4509-8a25-516358d3d83e-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:02.658526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.658440    7019 generic.go:356] "Generic (PLEG): container finished" podID="e6348ed9-1285-4509-8a25-516358d3d83e" containerID="ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d" exitCode=0
Nov 02 15:27:02.658526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.658488    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"
Nov 02 15:27:02.658792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.658529    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" event={"ID":"e6348ed9-1285-4509-8a25-516358d3d83e","Type":"ContainerDied","Data":"ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d"}
Nov 02 15:27:02.658792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.658583    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv" event={"ID":"e6348ed9-1285-4509-8a25-516358d3d83e","Type":"ContainerDied","Data":"c48384197c07fa3dd611ac12672e4755ba4eb21bd7ad8f8d678a82a6a0ff6e68"}
Nov 02 15:27:02.658792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.658614    7019 scope.go:122] "RemoveContainer" containerID="ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d"
Nov 02 15:27:02.660576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.660549    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" event={"ID":"ee941490-62a1-40fc-92f9-403df334d9bf","Type":"ContainerStarted","Data":"d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20"}
Nov 02 15:27:02.660721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.660602    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" event={"ID":"ee941490-62a1-40fc-92f9-403df334d9bf","Type":"ContainerStarted","Data":"aad775ecdcf0c18303fc73b6e88a9bd5d3bfb7112722bd3a357d9358f805cd31"}
Nov 02 15:27:02.704233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.704205    7019 scope.go:122] "RemoveContainer" containerID="ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d"
Nov 02 15:27:02.704588 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:27:02.704558    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d\": container with ID starting with ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d not found: ID does not exist" containerID="ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d"
Nov 02 15:27:02.704682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.704602    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d"} err="failed to get container status \"ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d\": rpc error: code = NotFound desc = could not find container \"ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d\": container with ID starting with ecd92beadd38ab86d258f94706d7faa92da327de8997360710eaaaa68a1a406d not found: ID does not exist"
Nov 02 15:27:02.728758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.728707    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podStartSLOduration=1.728693493 podStartE2EDuration="1.728693493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:02.72454714 +0000 UTC m=+1816.978500765" watchObservedRunningTime="2026-11-02 15:27:02.728693493 +0000 UTC m=+1816.982647119"
Nov 02 15:27:02.748886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.748850    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:02.776095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.776050    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"]
Nov 02 15:27:02.783153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:02.783122    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-knhdv"]
Nov 02 15:27:03.755079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:03.755047    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:03.863597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:03.863568    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" path="/var/lib/kubelet/pods/e6348ed9-1285-4509-8a25-516358d3d83e/volumes"
Nov 02 15:27:03.904554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:03.904518    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:04.679744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:04.679716    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:04.680975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:04.680951    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:27:05.498828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.498792    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"]
Nov 02 15:27:05.500501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.500475    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy"
Nov 02 15:27:05.500674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.500659    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy"
Nov 02 15:27:05.501033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.501019    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6348ed9-1285-4509-8a25-516358d3d83e" containerName="istio-proxy"
Nov 02 15:27:05.506962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.506939    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.513258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.511194    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"]
Nov 02 15:27:05.531020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.530983    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:27:05.531519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.531482    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:27:05.625503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.625470    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d765c83-2f44-450c-8347-5ada14347576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.625668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.625522    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8st\" (UniqueName: \"kubernetes.io/projected/1d765c83-2f44-450c-8347-5ada14347576-kube-api-access-bl8st\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.625715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.625694    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d765c83-2f44-450c-8347-5ada14347576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.719160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.719125    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:05.727563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.727535    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d765c83-2f44-450c-8347-5ada14347576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.746847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.727584    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8st\" (UniqueName: \"kubernetes.io/projected/1d765c83-2f44-450c-8347-5ada14347576-kube-api-access-bl8st\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.746847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.728328    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d765c83-2f44-450c-8347-5ada14347576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.746847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.728839    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d765c83-2f44-450c-8347-5ada14347576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.770965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.750170    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d765c83-2f44-450c-8347-5ada14347576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.783715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.782142    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8st\" (UniqueName: \"kubernetes.io/projected/1d765c83-2f44-450c-8347-5ada14347576-kube-api-access-bl8st\") pod \"maas-api-revoke-sub-8b034def-7wm9f\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:05.832698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:05.832628    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:06.148952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.148917    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"]
Nov 02 15:27:06.149123 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:27:06.149100    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d765c83_2f44_450c_8347_5ada14347576.slice/crio-f0c0806d9f77ab761dd752110e3fcf2b936205bafee451befafa3ea57f9688e0 WatchSource:0}: Error finding container f0c0806d9f77ab761dd752110e3fcf2b936205bafee451befafa3ea57f9688e0: Status 404 returned error can't find the container with id f0c0806d9f77ab761dd752110e3fcf2b936205bafee451befafa3ea57f9688e0
Nov 02 15:27:06.184580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.184554    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:06.363768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.363736    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:06.523288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.523257    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:27:06.540826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.540793    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:06.699382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.699342    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f" event={"ID":"1d765c83-2f44-450c-8347-5ada14347576","Type":"ContainerStarted","Data":"1086f9c471a3cfda49a9b6fbb26a94457613b7dd3915a3f7eb7a01562a539891"}
Nov 02 15:27:06.699627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.699407    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f" event={"ID":"1d765c83-2f44-450c-8347-5ada14347576","Type":"ContainerStarted","Data":"f0c0806d9f77ab761dd752110e3fcf2b936205bafee451befafa3ea57f9688e0"}
Nov 02 15:27:06.789520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:06.789424    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f" podStartSLOduration=1.789411618 podStartE2EDuration="1.789411618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:06.786832851 +0000 UTC m=+1821.040786475" watchObservedRunningTime="2026-11-02 15:27:06.789411618 +0000 UTC m=+1821.043365242"
Nov 02 15:27:07.711410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:07.711368    7019 generic.go:356] "Generic (PLEG): container finished" podID="1d765c83-2f44-450c-8347-5ada14347576" containerID="1086f9c471a3cfda49a9b6fbb26a94457613b7dd3915a3f7eb7a01562a539891" exitCode=0
Nov 02 15:27:07.711832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:07.711435    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f" event={"ID":"1d765c83-2f44-450c-8347-5ada14347576","Type":"ContainerDied","Data":"1086f9c471a3cfda49a9b6fbb26a94457613b7dd3915a3f7eb7a01562a539891"}
Nov 02 15:27:07.905457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:07.905417    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:08.602166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.602132    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:08.706529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.706490    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"]
Nov 02 15:27:08.711072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.711047    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:08.745511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.745473    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d35a86-dockercfg-p7mcw\""
Nov 02 15:27:08.746006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.745590    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d35a86\""
Nov 02 15:27:08.750662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.750616    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"]
Nov 02 15:27:08.768100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.768048    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:08.787217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.787180    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:08.787394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.787332    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdjfp\" (UniqueName: \"kubernetes.io/projected/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-kube-api-access-qdjfp\") pod \"payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:08.938635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.890977    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:08.939095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.928351    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:08.949022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:08.948903    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdjfp\" (UniqueName: \"kubernetes.io/projected/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-kube-api-access-qdjfp\") pod \"payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:09.034667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.030805    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdjfp\" (UniqueName: \"kubernetes.io/projected/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-kube-api-access-qdjfp\") pod \"payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:09.225570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.225543    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:09.326856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.326823    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:09.332125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.332080    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:09.360117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.360082    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d765c83-2f44-450c-8347-5ada14347576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d765c83-2f44-450c-8347-5ada14347576-maas-api-openshift-ca\") pod \"1d765c83-2f44-450c-8347-5ada14347576\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") "
Nov 02 15:27:09.361133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.360407    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d765c83-2f44-450c-8347-5ada14347576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d765c83-2f44-450c-8347-5ada14347576-maas-api-xks-ca\") pod \"1d765c83-2f44-450c-8347-5ada14347576\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") "
Nov 02 15:27:09.361133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.360467    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d765c83-2f44-450c-8347-5ada14347576-kube-api-access-bl8st\" (UniqueName: \"kubernetes.io/projected/1d765c83-2f44-450c-8347-5ada14347576-kube-api-access-bl8st\") pod \"1d765c83-2f44-450c-8347-5ada14347576\" (UID: \"1d765c83-2f44-450c-8347-5ada14347576\") "
Nov 02 15:27:09.370671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.365045    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d765c83-2f44-450c-8347-5ada14347576-maas-api-openshift-ca" pod "1d765c83-2f44-450c-8347-5ada14347576" (UID: "1d765c83-2f44-450c-8347-5ada14347576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:09.373542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.371590    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d765c83-2f44-450c-8347-5ada14347576-maas-api-xks-ca" pod "1d765c83-2f44-450c-8347-5ada14347576" (UID: "1d765c83-2f44-450c-8347-5ada14347576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:27:09.382677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.374890    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d765c83-2f44-450c-8347-5ada14347576-kube-api-access-bl8st" pod "1d765c83-2f44-450c-8347-5ada14347576" (UID: "1d765c83-2f44-450c-8347-5ada14347576"). InnerVolumeSpecName "kube-api-access-bl8st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:09.466670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.466573    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d765c83-2f44-450c-8347-5ada14347576-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:09.466670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.466603    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl8st\" (UniqueName: \"kubernetes.io/projected/1d765c83-2f44-450c-8347-5ada14347576-kube-api-access-bl8st\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:09.466670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.466618    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d765c83-2f44-450c-8347-5ada14347576-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:09.671461 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:27:09.671430    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ab4380_3dc2_49b4_9c9c_f6eba9a44a92.slice/crio-89af593817e89c93cd322310f6d5ceea88822e40b6c516c7d2f6ec466ab31903 WatchSource:0}: Error finding container 89af593817e89c93cd322310f6d5ceea88822e40b6c516c7d2f6ec466ab31903: Status 404 returned error can't find the container with id 89af593817e89c93cd322310f6d5ceea88822e40b6c516c7d2f6ec466ab31903
Nov 02 15:27:09.673926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.673896    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"]
Nov 02 15:27:09.754587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.754478    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"
Nov 02 15:27:09.755151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.754476    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f" event={"ID":"1d765c83-2f44-450c-8347-5ada14347576","Type":"ContainerDied","Data":"f0c0806d9f77ab761dd752110e3fcf2b936205bafee451befafa3ea57f9688e0"}
Nov 02 15:27:09.755151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.754705    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0c0806d9f77ab761dd752110e3fcf2b936205bafee451befafa3ea57f9688e0"
Nov 02 15:27:09.756428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.756404    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" event={"ID":"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92","Type":"ContainerStarted","Data":"89af593817e89c93cd322310f6d5ceea88822e40b6c516c7d2f6ec466ab31903"}
Nov 02 15:27:09.872068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:09.872032    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:10.031380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.031349    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:10.261746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.261714    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"]
Nov 02 15:27:10.262404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.262391    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1d765c83-2f44-450c-8347-5ada14347576" containerName="revoke-keys"
Nov 02 15:27:10.262447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.262405    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d765c83-2f44-450c-8347-5ada14347576" containerName="revoke-keys"
Nov 02 15:27:10.262556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.262546    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d765c83-2f44-450c-8347-5ada14347576" containerName="revoke-keys"
Nov 02 15:27:10.266460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.266436    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.298499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.298463    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cfc63a-openshift-default-dockercfg-gl7zv\""
Nov 02 15:27:10.303190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.303156    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"]
Nov 02 15:27:10.370670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.365296    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.381892    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istiod-ca-cert\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.381965    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrr24\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-kube-api-access-jrr24\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.382005    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-podinfo\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.382059    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-data\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.382104    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-token\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.382181    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-certs\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.382414    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-envoy\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.382521    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-socket\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.386674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.382642    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-credential-socket\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488437    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-credential-socket\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488554    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istiod-ca-cert\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488604    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrr24\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-kube-api-access-jrr24\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488637    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-podinfo\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488708    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-data\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488752    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-token\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488810    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-certs\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.488929    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-envoy\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.489016    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-socket\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.490710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.490513    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-credential-socket\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.498670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.492089    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-certs\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.498670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.498355    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-data\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.498930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.498817    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-socket\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.520671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.517233    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-envoy\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.520671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.518090    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istiod-ca-cert\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.541729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.540224    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-podinfo\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.569671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.563454    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrr24\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-kube-api-access-jrr24\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.594666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.591569    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-token\") pod \"e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") " pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:10.768290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.768249    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" event={"ID":"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92","Type":"ContainerStarted","Data":"479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd"}
Nov 02 15:27:10.768733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.768395    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:10.812065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.811950    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" podStartSLOduration=2.811933035 podStartE2EDuration="2.811933035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:10.81088954 +0000 UTC m=+1825.064843165" watchObservedRunningTime="2026-11-02 15:27:10.811933035 +0000 UTC m=+1825.065886661"
Nov 02 15:27:10.881793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:10.881752    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:11.087483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.084920    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:11.214889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.214850    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"]
Nov 02 15:27:11.215352 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:27:11.215323    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d85dc4_dad6_4112_a6d0_ca4f29074444.slice/crio-168c55230878287b07ec68b3ca8bcc37dba1dc7c1107e1f50f06308306505640 WatchSource:0}: Error finding container 168c55230878287b07ec68b3ca8bcc37dba1dc7c1107e1f50f06308306505640: Status 404 returned error can't find the container with id 168c55230878287b07ec68b3ca8bcc37dba1dc7c1107e1f50f06308306505640
Nov 02 15:27:11.218837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.218802    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:27:11.218928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.218913    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:27:11.218983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.218953    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:27:11.779625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.779588    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" event={"ID":"d6d85dc4-dad6-4112-a6d0-ca4f29074444","Type":"ContainerStarted","Data":"39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73"}
Nov 02 15:27:11.780168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.779663    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" event={"ID":"d6d85dc4-dad6-4112-a6d0-ca4f29074444","Type":"ContainerStarted","Data":"168c55230878287b07ec68b3ca8bcc37dba1dc7c1107e1f50f06308306505640"}
Nov 02 15:27:11.800720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.800641    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:11.841263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.841200    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podStartSLOduration=1.841179217 podStartE2EDuration="1.841179217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:11.840034642 +0000 UTC m=+1826.093988266" watchObservedRunningTime="2026-11-02 15:27:11.841179217 +0000 UTC m=+1826.095132843"
Nov 02 15:27:11.881901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:11.881868    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:12.888517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:12.888481    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:13.646642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:13.646607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:13.803222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:13.803186    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:13.804336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:13.804310    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:27:14.330810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:14.330772    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:14.529596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:14.529560    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:14.685525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:14.685487    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:15.413434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:15.413395    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:15.590837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:15.590801    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:16.605408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:16.605375    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:16.795548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:16.795152    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"]
Nov 02 15:27:16.809539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:16.809470    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:16.984269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:16.984235    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:17.507681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.507606    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"]
Nov 02 15:27:17.516316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.516287    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.530170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.530138    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"]
Nov 02 15:27:17.539752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.539726    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:27:17.539883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.539726    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:27:17.595656 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.595584    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.595932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.595708    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.595932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.595841    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p67hp\" (UniqueName: \"kubernetes.io/projected/8c1d3c7b-e5ae-4385-b38b-32efa518118c-kube-api-access-p67hp\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.663392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.663355    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:17.700349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.700312    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.700580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.700409    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.700580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.700530    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p67hp\" (UniqueName: \"kubernetes.io/projected/8c1d3c7b-e5ae-4385-b38b-32efa518118c-kube-api-access-p67hp\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.729738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.714015    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.740558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.740516    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.748161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.748128    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p67hp\" (UniqueName: \"kubernetes.io/projected/8c1d3c7b-e5ae-4385-b38b-32efa518118c-kube-api-access-p67hp\") pod \"maas-api-revoke-sub-d161ec86-l2nz2\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:17.833321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:17.833232    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:18.163340 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:27:18.163301    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1d3c7b_e5ae_4385_b38b_32efa518118c.slice/crio-f92edacf38d8872f5b9781b8a2318bd6931d2b67a3408e8f7cf9463d96435ff7 WatchSource:0}: Error finding container f92edacf38d8872f5b9781b8a2318bd6931d2b67a3408e8f7cf9463d96435ff7: Status 404 returned error can't find the container with id f92edacf38d8872f5b9781b8a2318bd6931d2b67a3408e8f7cf9463d96435ff7
Nov 02 15:27:18.163877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:18.163849    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"]
Nov 02 15:27:18.857110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:18.857034    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2" event={"ID":"8c1d3c7b-e5ae-4385-b38b-32efa518118c","Type":"ContainerStarted","Data":"d0ca25eea1e388ccce15e28211be1dd62cd76f1dd89927ee8f6e69f7c75b4f63"}
Nov 02 15:27:18.857110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:18.857075    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2" event={"ID":"8c1d3c7b-e5ae-4385-b38b-32efa518118c","Type":"ContainerStarted","Data":"f92edacf38d8872f5b9781b8a2318bd6931d2b67a3408e8f7cf9463d96435ff7"}
Nov 02 15:27:18.902917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:18.902853    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2" podStartSLOduration=1.902835315 podStartE2EDuration="1.902835315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:18.900112487 +0000 UTC m=+1833.154066122" watchObservedRunningTime="2026-11-02 15:27:18.902835315 +0000 UTC m=+1833.156788940"
Nov 02 15:27:19.366589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:19.366551    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:19.865201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:19.865168    7019 generic.go:356] "Generic (PLEG): container finished" podID="8c1d3c7b-e5ae-4385-b38b-32efa518118c" containerID="d0ca25eea1e388ccce15e28211be1dd62cd76f1dd89927ee8f6e69f7c75b4f63" exitCode=0
Nov 02 15:27:19.865616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:19.865217    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2" event={"ID":"8c1d3c7b-e5ae-4385-b38b-32efa518118c","Type":"ContainerDied","Data":"d0ca25eea1e388ccce15e28211be1dd62cd76f1dd89927ee8f6e69f7c75b4f63"}
Nov 02 15:27:20.499787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:20.499744    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:20.768429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:20.768313    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:21.064358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.064333    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:21.137673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.134753    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:21.185668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.176370    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-openshift-ca\") pod \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") "
Nov 02 15:27:21.185668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.176436    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-xks-ca\") pod \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") "
Nov 02 15:27:21.185668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.176490    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1d3c7b-e5ae-4385-b38b-32efa518118c-kube-api-access-p67hp\" (UniqueName: \"kubernetes.io/projected/8c1d3c7b-e5ae-4385-b38b-32efa518118c-kube-api-access-p67hp\") pod \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\" (UID: \"8c1d3c7b-e5ae-4385-b38b-32efa518118c\") "
Nov 02 15:27:21.185668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.185261    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-openshift-ca" pod "8c1d3c7b-e5ae-4385-b38b-32efa518118c" (UID: "8c1d3c7b-e5ae-4385-b38b-32efa518118c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:21.188389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.188353    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1d3c7b-e5ae-4385-b38b-32efa518118c-kube-api-access-p67hp" pod "8c1d3c7b-e5ae-4385-b38b-32efa518118c" (UID: "8c1d3c7b-e5ae-4385-b38b-32efa518118c"). InnerVolumeSpecName "kube-api-access-p67hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:21.191128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.191095    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-xks-ca" pod "8c1d3c7b-e5ae-4385-b38b-32efa518118c" (UID: "8c1d3c7b-e5ae-4385-b38b-32efa518118c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:27:21.279398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.279359    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:21.279398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.279399    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p67hp\" (UniqueName: \"kubernetes.io/projected/8c1d3c7b-e5ae-4385-b38b-32efa518118c-kube-api-access-p67hp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:21.279611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.279417    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1d3c7b-e5ae-4385-b38b-32efa518118c-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:21.783690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.783643    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:27:21.889571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.889535    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2" event={"ID":"8c1d3c7b-e5ae-4385-b38b-32efa518118c","Type":"ContainerDied","Data":"f92edacf38d8872f5b9781b8a2318bd6931d2b67a3408e8f7cf9463d96435ff7"}
Nov 02 15:27:21.889754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.889602    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f92edacf38d8872f5b9781b8a2318bd6931d2b67a3408e8f7cf9463d96435ff7"
Nov 02 15:27:21.889856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:21.889839    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"
Nov 02 15:27:22.712602 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:27:22.712558    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:27:22.744100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:22.744064    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:24.277118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:24.277076    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:25.890331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:25.890283    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:27.618856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:27.611857    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:28.726026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:28.725981    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:28.916095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:28.916064    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:29.450814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:29.450775    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:31.216116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:31.216075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:31.835107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:31.835058    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw" podUID="3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" containerName="main" containerID="cri-o://a4e65c781531684acd400c857df25c062be0d1cc514d107205fd2fd6567a39a7" gracePeriod=45
Nov 02 15:27:31.991294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:31.991049    7019 generic.go:356] "Generic (PLEG): container finished" podID="3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" containerID="a4e65c781531684acd400c857df25c062be0d1cc514d107205fd2fd6567a39a7" exitCode=0
Nov 02 15:27:31.991294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:31.991234    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw" event={"ID":"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e","Type":"ContainerDied","Data":"a4e65c781531684acd400c857df25c062be0d1cc514d107205fd2fd6567a39a7"}
Nov 02 15:27:32.163787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.163754    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:27:32.259217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.259177    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-kube-api-access-gqgd9\" (UniqueName: \"kubernetes.io/projected/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-kube-api-access-gqgd9\") pod \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") "
Nov 02 15:27:32.259750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.259380    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-model-cache\") pod \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") "
Nov 02 15:27:32.259750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.259418    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-dshm\") pod \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") "
Nov 02 15:27:32.259750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.259628    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tmp-dir\") pod \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") "
Nov 02 15:27:32.259750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.259704    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-model-cache" pod "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" (UID: "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:32.260011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.259884    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tmp-dir" pod "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" (UID: "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:32.260011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.259953    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tls-certs\") pod \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") "
Nov 02 15:27:32.260129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.260110    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-home\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-home\") pod \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\" (UID: \"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e\") "
Nov 02 15:27:32.260506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.260446    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-home" pod "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" (UID: "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:32.260919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.260900    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:32.261011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.260922    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:32.261011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.260934    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:32.262236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.262209    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-kube-api-access-gqgd9" pod "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" (UID: "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e"). InnerVolumeSpecName "kube-api-access-gqgd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:32.262343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.262276    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-dshm" pod "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" (UID: "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:27:32.262459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.262436    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tls-certs" pod "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" (UID: "3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:27:32.362309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.362276    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqgd9\" (UniqueName: \"kubernetes.io/projected/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-kube-api-access-gqgd9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:32.362309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.362308    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:32.362515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.362320    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:32.844815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:32.844782    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:33.012669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.012193    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:27:33.016669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.013491    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:33.070673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.070066    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw" event={"ID":"3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e","Type":"ContainerDied","Data":"7dd6d3c228df09dcc1903f6cb5e39d404507ef601a3d17860ff6dc019cd80d41"}
Nov 02 15:27:33.070673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.070128    7019 scope.go:122] "RemoveContainer" containerID="a4e65c781531684acd400c857df25c062be0d1cc514d107205fd2fd6567a39a7"
Nov 02 15:27:33.070673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.070335    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"
Nov 02 15:27:33.253224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.253194    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"]
Nov 02 15:27:33.258528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.258495    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a713aa/test-model-a713aa-kserve-66d6c565bc-p7bkw"]
Nov 02 15:27:33.863299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:33.863264    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" path="/var/lib/kubelet/pods/3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e/volumes"
Nov 02 15:27:34.564741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:34.564706    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:36.152246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:36.152208    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:36.309107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:36.309073    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:36.464936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:36.464845    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:37.909971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:37.909918    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:39.586082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:39.586036    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:40.016443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:40.016405    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:40.161662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:40.161614    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:41.280949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:41.280896    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:42.881835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:42.881794    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:43.407107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.407019    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:43.659990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.659906    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"]
Nov 02 15:27:43.661224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.661161    7019 cpu_manager.go:414] "containerMap: removing container" podUID="8c1d3c7b-e5ae-4385-b38b-32efa518118c" containerName="revoke-keys"
Nov 02 15:27:43.661224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.661221    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1d3c7b-e5ae-4385-b38b-32efa518118c" containerName="revoke-keys"
Nov 02 15:27:43.661421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.661306    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" containerName="main"
Nov 02 15:27:43.661421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.661314    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" containerName="main"
Nov 02 15:27:43.661525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.661485    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6ac155-b878-4a2e-a9a5-7e3a1c2d1d4e" containerName="main"
Nov 02 15:27:43.661525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.661507    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1d3c7b-e5ae-4385-b38b-32efa518118c" containerName="revoke-keys"
Nov 02 15:27:43.666129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.666103    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.692678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.691830    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:27:43.692678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.691959    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:27:43.707412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.707366    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.707617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.707595    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.707813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.707793    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5s6g\" (UniqueName: \"kubernetes.io/projected/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-kube-api-access-w5s6g\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.710184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.709935    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"]
Nov 02 15:27:43.809809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.809764    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.810020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.809926    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.810109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.810076    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5s6g\" (UniqueName: \"kubernetes.io/projected/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-kube-api-access-w5s6g\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.810671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.810623    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.812659 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.812616    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:43.836573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:43.836545    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5s6g\" (UniqueName: \"kubernetes.io/projected/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-kube-api-access-w5s6g\") pod \"maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:44.010934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:44.010850    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:44.320415 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:27:44.320378    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b8e6be_ddfe_42c4_848a_d698d9b9cd5c.slice/crio-bbdf278fe423c8f9b4e4782a8044d2dcf92c5c146ea01dba67ada1e3c7af5da5 WatchSource:0}: Error finding container bbdf278fe423c8f9b4e4782a8044d2dcf92c5c146ea01dba67ada1e3c7af5da5: Status 404 returned error can't find the container with id bbdf278fe423c8f9b4e4782a8044d2dcf92c5c146ea01dba67ada1e3c7af5da5
Nov 02 15:27:44.322016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:44.321986    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"]
Nov 02 15:27:44.717726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:44.717623    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:45.251443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:45.251401    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l" event={"ID":"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c","Type":"ContainerStarted","Data":"c81b6a6c4df93459a5bda1e200129f25c2efc964214c23413b4b545c9fc75b54"}
Nov 02 15:27:45.251875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:45.251478    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l" event={"ID":"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c","Type":"ContainerStarted","Data":"bbdf278fe423c8f9b4e4782a8044d2dcf92c5c146ea01dba67ada1e3c7af5da5"}
Nov 02 15:27:45.296696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:45.296611    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l" podStartSLOduration=2.296596355 podStartE2EDuration="2.296596355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:27:45.295895668 +0000 UTC m=+1859.549849291" watchObservedRunningTime="2026-11-02 15:27:45.296596355 +0000 UTC m=+1859.550550028"
Nov 02 15:27:46.260996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:46.260965    7019 generic.go:356] "Generic (PLEG): container finished" podID="55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" containerID="c81b6a6c4df93459a5bda1e200129f25c2efc964214c23413b4b545c9fc75b54" exitCode=0
Nov 02 15:27:46.261393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:46.261050    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l" event={"ID":"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c","Type":"ContainerDied","Data":"c81b6a6c4df93459a5bda1e200129f25c2efc964214c23413b4b545c9fc75b54"}
Nov 02 15:27:46.375808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:46.375764    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:47.463745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.463717    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:47.576524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.576441    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-xks-ca\") pod \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") "
Nov 02 15:27:47.576864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.576836    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-openshift-ca\") pod \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") "
Nov 02 15:27:47.576959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.576892    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-kube-api-access-w5s6g\" (UniqueName: \"kubernetes.io/projected/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-kube-api-access-w5s6g\") pod \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\" (UID: \"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c\") "
Nov 02 15:27:47.577237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.577208    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-openshift-ca" pod "55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" (UID: "55b8e6be-ddfe-42c4-848a-d698d9b9cd5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:27:47.577583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.577559    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:47.579166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.579128    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-xks-ca" pod "55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" (UID: "55b8e6be-ddfe-42c4-848a-d698d9b9cd5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:27:47.579360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.579331    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-kube-api-access-w5s6g" pod "55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" (UID: "55b8e6be-ddfe-42c4-848a-d698d9b9cd5c"). InnerVolumeSpecName "kube-api-access-w5s6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:27:47.618981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.618949    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:47.679419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.679381    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5s6g\" (UniqueName: \"kubernetes.io/projected/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-kube-api-access-w5s6g\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:47.679419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:47.679414    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:27:48.207744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:48.206697    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:48.280340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:48.280303    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"
Nov 02 15:27:48.280340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:48.280329    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l" event={"ID":"55b8e6be-ddfe-42c4-848a-d698d9b9cd5c","Type":"ContainerDied","Data":"bbdf278fe423c8f9b4e4782a8044d2dcf92c5c146ea01dba67ada1e3c7af5da5"}
Nov 02 15:27:48.280612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:48.280402    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbdf278fe423c8f9b4e4782a8044d2dcf92c5c146ea01dba67ada1e3c7af5da5"
Nov 02 15:27:49.773847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:49.773805    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:51.299960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:51.299920    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:51.439544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:51.439499    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:52.714120 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:27:52.714090    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:27:53.117735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:53.114300    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:54.184852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:54.184816    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"]
Nov 02 15:27:54.333974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:54.333941    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:54.484693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:54.484547    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:54.767754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:54.767641    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:56.368851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:56.368815    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:57.189250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:57.189209    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"]
Nov 02 15:27:57.189501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:57.189477    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy" containerID="cri-o://7eb6cdf4f1ecbc09a72f03e7129080d5e4fc2b2eccdeff88a6d336670730caec" gracePeriod=30
Nov 02 15:27:57.979307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:57.979272    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:59.185756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:59.185717    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" podUID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" containerName="payload-pre-processing" containerID="cri-o://eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702" gracePeriod=25
Nov 02 15:27:59.653012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:59.652978    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:27:59.797151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:27:59.797115    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:01.355040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:01.355006    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:01.505159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:01.505111    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:02.459865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.459792    7019 generic.go:356] "Generic (PLEG): container finished" podID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerID="7eb6cdf4f1ecbc09a72f03e7129080d5e4fc2b2eccdeff88a6d336670730caec" exitCode=0
Nov 02 15:28:02.459865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.459826    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" event={"ID":"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1","Type":"ContainerDied","Data":"7eb6cdf4f1ecbc09a72f03e7129080d5e4fc2b2eccdeff88a6d336670730caec"}
Nov 02 15:28:02.460273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.459883    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7" event={"ID":"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1","Type":"ContainerDied","Data":"2092244028d8d42d13535fbedb8ac8971512d830bc6254bd9718e9e2bdcc0814"}
Nov 02 15:28:02.460273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.459899    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2092244028d8d42d13535fbedb8ac8971512d830bc6254bd9718e9e2bdcc0814"
Nov 02 15:28:02.483026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.482998    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:28:02.560788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.560749    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-credential-socket\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.560943    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-certs\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.561012    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-socket\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.561083    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-token\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-token\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.561126    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-data\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.561183    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-kube-api-access-q85gz\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-kube-api-access-q85gz\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.561283    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istiod-ca-cert\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.561324    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-envoy\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.561537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.561475    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-podinfo\") pod \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\" (UID: \"fb3da149-f1cd-4386-9d8e-8bd2ec2122f1\") "
Nov 02 15:28:02.562798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.562059    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-credential-socket" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:02.562798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.562068    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-certs" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:02.562798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.562094    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-socket" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:02.562798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.562235    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-data" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:02.562798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.562497    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istiod-ca-cert" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:02.563520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.563498    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.563629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.563525    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.563629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.563540    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.563629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.563553    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.563629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.563566    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.565139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.565109    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-envoy" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:02.565288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.565238    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-token" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:02.565582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.565553    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-podinfo" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:28:02.566093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.566073    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-kube-api-access-q85gz" pod "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" (UID: "fb3da149-f1cd-4386-9d8e-8bd2ec2122f1"). InnerVolumeSpecName "kube-api-access-q85gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:02.668298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.668263    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.668298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.668297    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q85gz\" (UniqueName: \"kubernetes.io/projected/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-kube-api-access-q85gz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.668298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.668314    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.669088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.668328    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:02.997798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:02.994812    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:03.467327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:03.467296    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"
Nov 02 15:28:03.503092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:03.503053    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" podUID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:28:03.536905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:03.536869    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"]
Nov 02 15:28:03.540790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:03.540764    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-7ecfe1-openshift-default-6f756657bc-f76f7"]
Nov 02 15:28:03.863878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:03.863848    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" path="/var/lib/kubelet/pods/fb3da149-f1cd-4386-9d8e-8bd2ec2122f1/volumes"
Nov 02 15:28:04.676822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:04.676779    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:05.424220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.424181    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"]
Nov 02 15:28:05.424453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.424428    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl" podUID="005828cd-e88b-4f49-85fe-957713a31b47" containerName="discovery" containerID="cri-o://27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482" gracePeriod=30
Nov 02 15:28:05.733686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.733643    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:28:05.839168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.839132    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-dns-cert\") pod \"005828cd-e88b-4f49-85fe-957713a31b47\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") "
Nov 02 15:28:05.839614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.839196    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-kube-api-access-cm54j\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-kube-api-access-cm54j\") pod \"005828cd-e88b-4f49-85fe-957713a31b47\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") "
Nov 02 15:28:05.839614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.839268    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-istio-token\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-istio-token\") pod \"005828cd-e88b-4f49-85fe-957713a31b47\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") "
Nov 02 15:28:05.839614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.839310    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-kubeconfig\") pod \"005828cd-e88b-4f49-85fe-957713a31b47\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") "
Nov 02 15:28:05.839614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.839360    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-ca-configmap\") pod \"005828cd-e88b-4f49-85fe-957713a31b47\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") "
Nov 02 15:28:05.839614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.839569    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-cacerts\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-cacerts\") pod \"005828cd-e88b-4f49-85fe-957713a31b47\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") "
Nov 02 15:28:05.839981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.839702    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005828cd-e88b-4f49-85fe-957713a31b47-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005828cd-e88b-4f49-85fe-957713a31b47-local-certs\") pod \"005828cd-e88b-4f49-85fe-957713a31b47\" (UID: \"005828cd-e88b-4f49-85fe-957713a31b47\") "
Nov 02 15:28:05.840864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.840838    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-ca-configmap" pod "005828cd-e88b-4f49-85fe-957713a31b47" (UID: "005828cd-e88b-4f49-85fe-957713a31b47"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:05.853333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.853284    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005828cd-e88b-4f49-85fe-957713a31b47-local-certs" pod "005828cd-e88b-4f49-85fe-957713a31b47" (UID: "005828cd-e88b-4f49-85fe-957713a31b47"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:05.853470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.853366    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-kubeconfig" pod "005828cd-e88b-4f49-85fe-957713a31b47" (UID: "005828cd-e88b-4f49-85fe-957713a31b47"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:28:05.854698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.854080    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-cacerts" pod "005828cd-e88b-4f49-85fe-957713a31b47" (UID: "005828cd-e88b-4f49-85fe-957713a31b47"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:28:05.861144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.861116    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-istio-token" pod "005828cd-e88b-4f49-85fe-957713a31b47" (UID: "005828cd-e88b-4f49-85fe-957713a31b47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:05.865863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.865831    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-kube-api-access-cm54j" pod "005828cd-e88b-4f49-85fe-957713a31b47" (UID: "005828cd-e88b-4f49-85fe-957713a31b47"). InnerVolumeSpecName "kube-api-access-cm54j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:05.868115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.868086    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-dns-cert" pod "005828cd-e88b-4f49-85fe-957713a31b47" (UID: "005828cd-e88b-4f49-85fe-957713a31b47"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:28:05.942493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.942451    7019 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005828cd-e88b-4f49-85fe-957713a31b47-local-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:05.942493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.942493    7019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-dns-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:05.942754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.942508    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm54j\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-kube-api-access-cm54j\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:05.942754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.942522    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005828cd-e88b-4f49-85fe-957713a31b47-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:05.942754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.942538    7019 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-istio-kubeconfig\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:05.942754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.942550    7019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005828cd-e88b-4f49-85fe-957713a31b47-istio-csr-ca-configmap\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:05.942754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:05.942562    7019 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005828cd-e88b-4f49-85fe-957713a31b47-cacerts\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:06.504545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.504508    7019 generic.go:356] "Generic (PLEG): container finished" podID="005828cd-e88b-4f49-85fe-957713a31b47" containerID="27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482" exitCode=0
Nov 02 15:28:06.504765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.504561    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl" event={"ID":"005828cd-e88b-4f49-85fe-957713a31b47","Type":"ContainerDied","Data":"27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482"}
Nov 02 15:28:06.504765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.504595    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl" event={"ID":"005828cd-e88b-4f49-85fe-957713a31b47","Type":"ContainerDied","Data":"790014dbb571f3781b8f41508e59996d41f0552c11e825786a2f5f9f785c055b"}
Nov 02 15:28:06.504765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.504613    7019 scope.go:122] "RemoveContainer" containerID="27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482"
Nov 02 15:28:06.504765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.504560    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"
Nov 02 15:28:06.537202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.537176    7019 scope.go:122] "RemoveContainer" containerID="27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482"
Nov 02 15:28:06.537600 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:28:06.537569    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482\": container with ID starting with 27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482 not found: ID does not exist" containerID="27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482"
Nov 02 15:28:06.537775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.537611    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482"} err="failed to get container status \"27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482\": rpc error: code = NotFound desc = could not find container \"27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482\": container with ID starting with 27f533de0315f1ba238f61df2e1d410bb1f72bb1cf58f73b2a5996b57765f482 not found: ID does not exist"
Nov 02 15:28:06.574081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.574043    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:06.575007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.574984    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"]
Nov 02 15:28:06.589570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:06.589461    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-dkghl"]
Nov 02 15:28:07.864640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:07.864599    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005828cd-e88b-4f49-85fe-957713a31b47" path="/var/lib/kubelet/pods/005828cd-e88b-4f49-85fe-957713a31b47/volumes"
Nov 02 15:28:08.514671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:08.509184    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:08.832732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:08.832578    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:09.283921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.283888    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:09.675967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.675931    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"]
Nov 02 15:28:09.677339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.677308    7019 cpu_manager.go:414] "containerMap: removing container" podUID="55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" containerName="revoke-keys"
Nov 02 15:28:09.677507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.677493    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" containerName="revoke-keys"
Nov 02 15:28:09.677640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.677627    7019 cpu_manager.go:414] "containerMap: removing container" podUID="005828cd-e88b-4f49-85fe-957713a31b47" containerName="discovery"
Nov 02 15:28:09.677786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.677771    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005828cd-e88b-4f49-85fe-957713a31b47" containerName="discovery"
Nov 02 15:28:09.677915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.677904    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy"
Nov 02 15:28:09.677996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.677986    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy"
Nov 02 15:28:09.678266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.678251    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="005828cd-e88b-4f49-85fe-957713a31b47" containerName="discovery"
Nov 02 15:28:09.678400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.678369    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3da149-f1cd-4386-9d8e-8bd2ec2122f1" containerName="istio-proxy"
Nov 02 15:28:09.678499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.678488    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" containerName="revoke-keys"
Nov 02 15:28:09.682956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.682935    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.725892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.725853    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"]
Nov 02 15:28:09.738850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.738820    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-31c816-openshift-default-dockercfg-mmnpj\""
Nov 02 15:28:09.766725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.766689    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:09.902671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.902453    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-certs\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.902671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.902565    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-podinfo\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.902671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.902607    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-data\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.954789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.935766    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-socket\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.954789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.935823    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-envoy\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.954789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.935855    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqhn6\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-kube-api-access-dqhn6\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.954789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.935919    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-token\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.954789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.936076    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-credential-socket\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:09.954789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:09.936103    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istiod-ca-cert\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.042274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.042231    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-socket\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.042546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.042527    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-envoy\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.042696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.042665    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqhn6\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-kube-api-access-dqhn6\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.042857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.042842    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-token\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.043073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.043058    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-credential-socket\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.043184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.043171    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istiod-ca-cert\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.043485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.043458    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-certs\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.043680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.043664    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-podinfo\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.043804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.043790    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-data\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.057375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.044358    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-data\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.058601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.058257    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-socket\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.059042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.059021    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-credential-socket\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.060530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.059443    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-certs\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.060676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.060131    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istiod-ca-cert\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.088755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.087193    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqhn6\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-kube-api-access-dqhn6\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.088755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.087807    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-token\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.095595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.091233    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-envoy\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.104718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.100174    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-podinfo\") pod \"e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") " pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.143536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.138835    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:10.357426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.357335    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:10.681230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.681191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:10.705052 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:28:10.705021    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9b3c83_b754_4bbe_94fa_8036234b0e65.slice/crio-0e35bd228b432cef48795c748ef67922f1ebe4f5c396ce741b084ff9df66db58 WatchSource:0}: Error finding container 0e35bd228b432cef48795c748ef67922f1ebe4f5c396ce741b084ff9df66db58: Status 404 returned error can't find the container with id 0e35bd228b432cef48795c748ef67922f1ebe4f5c396ce741b084ff9df66db58
Nov 02 15:28:10.707489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.707451    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"]
Nov 02 15:28:10.708589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.708547    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:10.708700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.708623    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:10.708700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:10.708672    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:11.556721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:11.556680    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" event={"ID":"ab9b3c83-b754-4bbe-94fa-8036234b0e65","Type":"ContainerStarted","Data":"89deddf99ecfba4fdcef094eff85129c2a1abd5d4be6a7f6b41c010a987b1d83"}
Nov 02 15:28:11.557227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:11.556747    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" event={"ID":"ab9b3c83-b754-4bbe-94fa-8036234b0e65","Type":"ContainerStarted","Data":"0e35bd228b432cef48795c748ef67922f1ebe4f5c396ce741b084ff9df66db58"}
Nov 02 15:28:11.630484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:11.630420    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podStartSLOduration=2.63040101 podStartE2EDuration="2.63040101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:11.617195446 +0000 UTC m=+1885.871149070" watchObservedRunningTime="2026-11-02 15:28:11.63040101 +0000 UTC m=+1885.884354634"
Nov 02 15:28:12.300946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:12.300902    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:12.358042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:12.358009    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:12.365994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:12.365963    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:12.565026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:12.564931    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:12.566021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:12.565997    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:28:13.504275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:13.504231    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" podUID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:28:14.076209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:14.076170    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:14.250669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:14.247280    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:15.786609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.786525    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:15.918228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.918189    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"]
Nov 02 15:28:15.922886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.922864    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:15.929009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.928979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:15.939582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.939557    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-31c816\""
Nov 02 15:28:15.952028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.951995    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-31c816-dockercfg-lsm9r\""
Nov 02 15:28:15.958047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.958018    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"]
Nov 02 15:28:15.958643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.958611    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c471f-15dc-47a0-b7d3-f10557a08f0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:15.958871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:15.958840    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcxt\" (UniqueName: \"kubernetes.io/projected/143c471f-15dc-47a0-b7d3-f10557a08f0a-kube-api-access-jkcxt\") pod \"payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:16.089731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.086690    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c471f-15dc-47a0-b7d3-f10557a08f0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:16.089731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.086891    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcxt\" (UniqueName: \"kubernetes.io/projected/143c471f-15dc-47a0-b7d3-f10557a08f0a-kube-api-access-jkcxt\") pod \"payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:16.089731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.088323    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c471f-15dc-47a0-b7d3-f10557a08f0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:16.217881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.210225    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:16.243672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.239271    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcxt\" (UniqueName: \"kubernetes.io/projected/143c471f-15dc-47a0-b7d3-f10557a08f0a-kube-api-access-jkcxt\") pod \"payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:16.255111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.254440    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:16.355148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.345781    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"]
Nov 02 15:28:16.367426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.367382    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:16.396293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.396260    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"]
Nov 02 15:28:16.400880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.400600    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9009810-8214-4b24-bbad-e86b2575ac41-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:16.401112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.401089    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpv2\" (UniqueName: \"kubernetes.io/projected/c9009810-8214-4b24-bbad-e86b2575ac41-kube-api-access-8xpv2\") pod \"payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:16.502411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.502369    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9009810-8214-4b24-bbad-e86b2575ac41-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:16.502628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.502417    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpv2\" (UniqueName: \"kubernetes.io/projected/c9009810-8214-4b24-bbad-e86b2575ac41-kube-api-access-8xpv2\") pod \"payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:16.503277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.503251    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9009810-8214-4b24-bbad-e86b2575ac41-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:16.527373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.527348    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpv2\" (UniqueName: \"kubernetes.io/projected/c9009810-8214-4b24-bbad-e86b2575ac41-kube-api-access-8xpv2\") pod \"payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") " pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:16.670293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.670259    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"]
Nov 02 15:28:16.700048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:16.700023    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:17.012018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.011980    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"]
Nov 02 15:28:17.013660 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:28:17.013616    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9009810_8214_4b24_bbad_e86b2575ac41.slice/crio-2d14e3cffe166def85100cf88769a53a10d5ab58c78e9a8e60b296f9f454a9a9 WatchSource:0}: Error finding container 2d14e3cffe166def85100cf88769a53a10d5ab58c78e9a8e60b296f9f454a9a9: Status 404 returned error can't find the container with id 2d14e3cffe166def85100cf88769a53a10d5ab58c78e9a8e60b296f9f454a9a9
Nov 02 15:28:17.610579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.610539    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" event={"ID":"143c471f-15dc-47a0-b7d3-f10557a08f0a","Type":"ContainerStarted","Data":"37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91"}
Nov 02 15:28:17.610837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.610604    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" event={"ID":"143c471f-15dc-47a0-b7d3-f10557a08f0a","Type":"ContainerStarted","Data":"a6fbbc1d4be93666fd6c166a6f27928f5ad440d9ec4d55cbd50f57223d1bb487"}
Nov 02 15:28:17.610919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.610888    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:17.612786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.612754    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx" event={"ID":"c9009810-8214-4b24-bbad-e86b2575ac41","Type":"ContainerStarted","Data":"e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925"}
Nov 02 15:28:17.612931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.612820    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:17.612931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.612844    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx" event={"ID":"c9009810-8214-4b24-bbad-e86b2575ac41","Type":"ContainerStarted","Data":"2d14e3cffe166def85100cf88769a53a10d5ab58c78e9a8e60b296f9f454a9a9"}
Nov 02 15:28:17.655209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.655157    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" podStartSLOduration=2.655142819 podStartE2EDuration="2.655142819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:17.65295055 +0000 UTC m=+1891.906904185" watchObservedRunningTime="2026-11-02 15:28:17.655142819 +0000 UTC m=+1891.909096497"
Nov 02 15:28:17.703611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:17.703527    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx" podStartSLOduration=2.703508652 podStartE2EDuration="2.703508652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:17.698168482 +0000 UTC m=+1891.952122108" watchObservedRunningTime="2026-11-02 15:28:17.703508652 +0000 UTC m=+1891.957462276"
Nov 02 15:28:18.162596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:18.162558    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:19.497543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.497512    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:28:19.553461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.553431    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-kube-api-access-84q79\" (UniqueName: \"kubernetes.io/projected/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-kube-api-access-84q79\") pod \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") "
Nov 02 15:28:19.553664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.553576    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-plugins-config-volume\") pod \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\" (UID: \"37a52df5-81c4-47b7-9d9a-fb632f71ceb8\") "
Nov 02 15:28:19.553994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.553964    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-plugins-config-volume" pod "37a52df5-81c4-47b7-9d9a-fb632f71ceb8" (UID: "37a52df5-81c4-47b7-9d9a-fb632f71ceb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:19.554506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.554481    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:19.555860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.555831    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-kube-api-access-84q79" pod "37a52df5-81c4-47b7-9d9a-fb632f71ceb8" (UID: "37a52df5-81c4-47b7-9d9a-fb632f71ceb8"). InnerVolumeSpecName "kube-api-access-84q79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:19.629640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.629608    7019 generic.go:356] "Generic (PLEG): container finished" podID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" containerID="eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702" exitCode=0
Nov 02 15:28:19.629852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.629675    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"
Nov 02 15:28:19.629852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.629683    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" event={"ID":"37a52df5-81c4-47b7-9d9a-fb632f71ceb8","Type":"ContainerDied","Data":"eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702"}
Nov 02 15:28:19.629852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.629743    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw" event={"ID":"37a52df5-81c4-47b7-9d9a-fb632f71ceb8","Type":"ContainerDied","Data":"492f6f22c2183b884ac33a62e19b24dceb5af4c50f8097971c578bb8a8c74282"}
Nov 02 15:28:19.629852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.629766    7019 scope.go:122] "RemoveContainer" containerID="eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702"
Nov 02 15:28:19.640861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.640786    7019 scope.go:122] "RemoveContainer" containerID="eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702"
Nov 02 15:28:19.641082 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:28:19.641059    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702\": container with ID starting with eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702 not found: ID does not exist" containerID="eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702"
Nov 02 15:28:19.641136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.641090    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702"} err="failed to get container status \"eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702\": rpc error: code = NotFound desc = could not find container \"eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702\": container with ID starting with eed06a24456d8f868683f587f4e3d045b42866c286967f6b9725eb0194296702 not found: ID does not exist"
Nov 02 15:28:19.655512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.655485    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84q79\" (UniqueName: \"kubernetes.io/projected/37a52df5-81c4-47b7-9d9a-fb632f71ceb8-kube-api-access-84q79\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:19.686244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.686208    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"]
Nov 02 15:28:19.689595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.689571    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a713aa-5745475cc9-jgkfw"]
Nov 02 15:28:19.863898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.863872    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" path="/var/lib/kubelet/pods/37a52df5-81c4-47b7-9d9a-fb632f71ceb8/volumes"
Nov 02 15:28:19.898415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:19.898377    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:21.553793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:21.553487    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:22.709850 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:28:22.709813    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:28:23.090998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:23.090879    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:24.848721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:24.844411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:26.344165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:26.343833    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"]
Nov 02 15:28:26.344165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:26.344067    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc" podUID="d5302cfd-d1f6-4310-ad41-c1cce3762503" containerName="maas-api" containerID="cri-o://71a5bf4ac326bc498ed7ce1b5a9f5ec74c8b18cea2d2c3188f270101b967ac4f" gracePeriod=30
Nov 02 15:28:26.501572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:26.501535    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:27.033944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.002259    7019 generic.go:356] "Generic (PLEG): container finished" podID="d5302cfd-d1f6-4310-ad41-c1cce3762503" containerID="71a5bf4ac326bc498ed7ce1b5a9f5ec74c8b18cea2d2c3188f270101b967ac4f" exitCode=0
Nov 02 15:28:27.033944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.002349    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc" event={"ID":"d5302cfd-d1f6-4310-ad41-c1cce3762503","Type":"ContainerDied","Data":"71a5bf4ac326bc498ed7ce1b5a9f5ec74c8b18cea2d2c3188f270101b967ac4f"}
Nov 02 15:28:27.033944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.005157    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:27.279671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.276120    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:27.366130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.349146    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"]
Nov 02 15:28:27.406879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.401954    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a713aa-s4r7l"]
Nov 02 15:28:27.435117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.435083    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:27.452854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.452568    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:28:27.509928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.509888    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-maas-api-tls\") pod \"d5302cfd-d1f6-4310-ad41-c1cce3762503\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") "
Nov 02 15:28:27.510745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.510719    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5302cfd-d1f6-4310-ad41-c1cce3762503-kube-api-access-csfph\" (UniqueName: \"kubernetes.io/projected/d5302cfd-d1f6-4310-ad41-c1cce3762503-kube-api-access-csfph\") pod \"d5302cfd-d1f6-4310-ad41-c1cce3762503\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") "
Nov 02 15:28:27.510940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.510924    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5302cfd-d1f6-4310-ad41-c1cce3762503-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5302cfd-d1f6-4310-ad41-c1cce3762503-openshift-service-ca\") pod \"d5302cfd-d1f6-4310-ad41-c1cce3762503\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") "
Nov 02 15:28:27.511147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.511132    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-metrics-certs\") pod \"d5302cfd-d1f6-4310-ad41-c1cce3762503\" (UID: \"d5302cfd-d1f6-4310-ad41-c1cce3762503\") "
Nov 02 15:28:27.512446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.512407    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5302cfd-d1f6-4310-ad41-c1cce3762503-openshift-service-ca" pod "d5302cfd-d1f6-4310-ad41-c1cce3762503" (UID: "d5302cfd-d1f6-4310-ad41-c1cce3762503"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:27.514804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.514765    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-metrics-certs" pod "d5302cfd-d1f6-4310-ad41-c1cce3762503" (UID: "d5302cfd-d1f6-4310-ad41-c1cce3762503"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:28:27.515298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.515266    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5302cfd-d1f6-4310-ad41-c1cce3762503-kube-api-access-csfph" pod "d5302cfd-d1f6-4310-ad41-c1cce3762503" (UID: "d5302cfd-d1f6-4310-ad41-c1cce3762503"). InnerVolumeSpecName "kube-api-access-csfph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:27.524575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.524525    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-maas-api-tls" pod "d5302cfd-d1f6-4310-ad41-c1cce3762503" (UID: "d5302cfd-d1f6-4310-ad41-c1cce3762503"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:28:27.632565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.622630    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfph\" (UniqueName: \"kubernetes.io/projected/d5302cfd-d1f6-4310-ad41-c1cce3762503-kube-api-access-csfph\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:27.632565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.626807    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5302cfd-d1f6-4310-ad41-c1cce3762503-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:27.632565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.626846    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:27.632565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.626861    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5302cfd-d1f6-4310-ad41-c1cce3762503-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:27.692457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.692273    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:27.864067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:27.864029    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b8e6be-ddfe-42c4-848a-d698d9b9cd5c" path="/var/lib/kubelet/pods/55b8e6be-ddfe-42c4-848a-d698d9b9cd5c/volumes"
Nov 02 15:28:28.033975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:28.032958    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc" event={"ID":"d5302cfd-d1f6-4310-ad41-c1cce3762503","Type":"ContainerDied","Data":"a8775ee911d624b5b43afed1e3e772a8513037031f04b820672e1d8b235f716e"}
Nov 02 15:28:28.033975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:28.033033    7019 scope.go:122] "RemoveContainer" containerID="71a5bf4ac326bc498ed7ce1b5a9f5ec74c8b18cea2d2c3188f270101b967ac4f"
Nov 02 15:28:28.033975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:28.033193    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"
Nov 02 15:28:28.125711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:28.123771    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"]
Nov 02 15:28:28.127859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:28.127825    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a713aa-58f774c4db-kzslc"]
Nov 02 15:28:28.624395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:28.624363    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:28:28.624827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:28.624504    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:28:29.035516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:29.035404    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:29.631674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:29.626268    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:29.882146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:29.882064    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5302cfd-d1f6-4310-ad41-c1cce3762503" path="/var/lib/kubelet/pods/d5302cfd-d1f6-4310-ad41-c1cce3762503/volumes"
Nov 02 15:28:30.091477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.091385    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:30.556732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.553180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:30.556988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.556963    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"]
Nov 02 15:28:30.561738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.558187    7019 cpu_manager.go:414] "containerMap: removing container" podUID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" containerName="payload-pre-processing"
Nov 02 15:28:30.561948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.561927    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" containerName="payload-pre-processing"
Nov 02 15:28:30.562087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.562074    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d5302cfd-d1f6-4310-ad41-c1cce3762503" containerName="maas-api"
Nov 02 15:28:30.562177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.562166    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5302cfd-d1f6-4310-ad41-c1cce3762503" containerName="maas-api"
Nov 02 15:28:30.562581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.562559    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a52df5-81c4-47b7-9d9a-fb632f71ceb8" containerName="payload-pre-processing"
Nov 02 15:28:30.562756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.562725    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5302cfd-d1f6-4310-ad41-c1cce3762503" containerName="maas-api"
Nov 02 15:28:30.656723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.650588    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"]
Nov 02 15:28:30.668676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.661217    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.686670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.683088    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2e4ebc-openshift-default-dockercfg-sxghc\""
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757175    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-envoy\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757239    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29329576-824e-44bb-a825-b93ab7391426-istio-podinfo\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757319    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsqr\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-kube-api-access-tjsqr\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757474    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-socket\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757500    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-data\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757526    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29329576-824e-44bb-a825-b93ab7391426-istiod-ca-cert\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757623    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-certs\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757688    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-credential-socket\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.773672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.757869    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-istio-token\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.880689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.880588    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:30.882561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.882522    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-istio-token\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.898056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.898025    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-envoy\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.898277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.898258    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29329576-824e-44bb-a825-b93ab7391426-istio-podinfo\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.898527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.898502    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsqr\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-kube-api-access-tjsqr\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.898834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.898817    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-socket\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.899040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.899024    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-data\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.899165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.899149    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29329576-824e-44bb-a825-b93ab7391426-istiod-ca-cert\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.899377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.899358    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-certs\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.899555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.899527    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-credential-socket\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.905977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.905189    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-socket\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.924190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.913484    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-envoy\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.924190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.916037    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-data\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.924190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.916917    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29329576-824e-44bb-a825-b93ab7391426-istiod-ca-cert\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.924190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.917239    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-certs\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.924190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.919792    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-credential-socket\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.954941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.944302    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29329576-824e-44bb-a825-b93ab7391426-istio-podinfo\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.970675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.962760    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-istio-token\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:30.975667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:30.971258    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsqr\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-kube-api-access-tjsqr\") pod \"e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") " pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:31.033257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:31.032966    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:31.085620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:31.056513    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:31.961816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:31.961767    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:31.962355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:31.962325    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:31.962489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:31.962372    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:31.964108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:31.964086    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"]
Nov 02 15:28:32.142330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:32.142293    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" event={"ID":"29329576-824e-44bb-a825-b93ab7391426","Type":"ContainerStarted","Data":"a043574f70a187c5a79a10fbf4b860579db6c7b6f23e294f3ae8edbf226709ed"}
Nov 02 15:28:32.423157 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:32.416411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:33.153113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:33.153019    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" event={"ID":"29329576-824e-44bb-a825-b93ab7391426","Type":"ContainerStarted","Data":"592225f8325dfea14ba8db1eced35d4fe12af35bbfd52a0b0db67c4cd5ee542f"}
Nov 02 15:28:33.221688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:33.221596    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podStartSLOduration=3.221582112 podStartE2EDuration="3.221582112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:33.219100294 +0000 UTC m=+1907.473053918" watchObservedRunningTime="2026-11-02 15:28:33.221582112 +0000 UTC m=+1907.475535735"
Nov 02 15:28:34.033638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.033595    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:34.042215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.042182    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:34.105669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.102297    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:34.180670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.168083    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:34.180670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.172980    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:28:34.250565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.250397    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:34.410003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.409918    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:34.535578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.535541    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"]
Nov 02 15:28:34.535862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.535834    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy" containerID="cri-o://73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493" gracePeriod=30
Nov 02 15:28:34.590837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:34.590639    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:35.099196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.095662    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"]
Nov 02 15:28:35.104232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.104205    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.106739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.106714    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"]
Nov 02 15:28:35.122059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.122025    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:28:35.122731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.122712    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:28:35.181565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.181280    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbqd\" (UniqueName: \"kubernetes.io/projected/91b85840-3a63-4c38-87ac-94d843f788d4-kube-api-access-mwbqd\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.181565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.181332    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.181565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.181419    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.288696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.285803    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbqd\" (UniqueName: \"kubernetes.io/projected/91b85840-3a63-4c38-87ac-94d843f788d4-kube-api-access-mwbqd\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.288696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.285854    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.288696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.285998    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.288696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.287983    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.289986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.289956    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.314716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.314602    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbqd\" (UniqueName: \"kubernetes.io/projected/91b85840-3a63-4c38-87ac-94d843f788d4-kube-api-access-mwbqd\") pod \"maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.428087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.427770    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:35.486134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.485831    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:28:35.486134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.485900    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:28:35.699805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.699727    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:35.768074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.768033    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"]
Nov 02 15:28:35.871791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:35.871756    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:36.054940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:36.054797    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:36.205355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:36.205313    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4" event={"ID":"91b85840-3a63-4c38-87ac-94d843f788d4","Type":"ContainerStarted","Data":"07258977fdbd253fd7f0b13189c9d3837e99376df3a9a3a5b9869e57f6e490a4"}
Nov 02 15:28:37.218522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.218436    7019 generic.go:356] "Generic (PLEG): container finished" podID="91b85840-3a63-4c38-87ac-94d843f788d4" containerID="4e29bfbd547b4658d0d2f76afdf2aded3803390999a6a549023abc8405aef219" exitCode=0
Nov 02 15:28:37.218969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.218529    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4" event={"ID":"91b85840-3a63-4c38-87ac-94d843f788d4","Type":"ContainerDied","Data":"4e29bfbd547b4658d0d2f76afdf2aded3803390999a6a549023abc8405aef219"}
Nov 02 15:28:37.406419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.406232    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:37.538909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.538828    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"]
Nov 02 15:28:37.542538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.542512    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:37.563631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.563552    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2e4ebc-dockercfg-bz8q5\""
Nov 02 15:28:37.563781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.563667    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2e4ebc\""
Nov 02 15:28:37.588672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.571105    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"]
Nov 02 15:28:37.588672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.578145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:37.638052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.637998    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bzx\" (UniqueName: \"kubernetes.io/projected/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-kube-api-access-m4bzx\") pod \"payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:37.638255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.638087    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:37.855210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.845770    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bzx\" (UniqueName: \"kubernetes.io/projected/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-kube-api-access-m4bzx\") pod \"payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:37.855210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.845873    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:37.855210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.847695    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:37.895389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.895160    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bzx\" (UniqueName: \"kubernetes.io/projected/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-kube-api-access-m4bzx\") pod \"payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:37.993708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:37.991494    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:38.025681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.024508    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"]
Nov 02 15:28:38.043865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.042906    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"]
Nov 02 15:28:38.043865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.043015    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.156189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.156148    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:38.156818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.156794    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfwj\" (UniqueName: \"kubernetes.io/projected/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-kube-api-access-xkfwj\") pod \"payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.156924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.156868    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.232454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.232369    7019 generic.go:356] "Generic (PLEG): container finished" podID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerID="7faaeeed3a4c74c7a409a2e0362957607fc832aa300fee7a914f912bfd78793f" exitCode=0
Nov 02 15:28:38.233140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.232637    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" event={"ID":"d8db75dc-11e7-4eeb-9c6f-f3128863a73a","Type":"ContainerDied","Data":"7faaeeed3a4c74c7a409a2e0362957607fc832aa300fee7a914f912bfd78793f"}
Nov 02 15:28:38.233273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.233162    7019 scope.go:122] "RemoveContainer" containerID="7faaeeed3a4c74c7a409a2e0362957607fc832aa300fee7a914f912bfd78793f"
Nov 02 15:28:38.236795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.236755    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:38.236914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.236892    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:38.237024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.236974    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:38.268750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.265826    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfwj\" (UniqueName: \"kubernetes.io/projected/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-kube-api-access-xkfwj\") pod \"payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.268750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.265998    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.268750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.267808    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.319832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.319370    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfwj\" (UniqueName: \"kubernetes.io/projected/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-kube-api-access-xkfwj\") pod \"payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.363777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.362158    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:38.516042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.515418    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"]
Nov 02 15:28:38.639686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.638302    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:38.716634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.716594    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-xks-ca\") pod \"91b85840-3a63-4c38-87ac-94d843f788d4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") "
Nov 02 15:28:38.716873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.716851    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b85840-3a63-4c38-87ac-94d843f788d4-kube-api-access-mwbqd\" (UniqueName: \"kubernetes.io/projected/91b85840-3a63-4c38-87ac-94d843f788d4-kube-api-access-mwbqd\") pod \"91b85840-3a63-4c38-87ac-94d843f788d4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") "
Nov 02 15:28:38.716962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.716938    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-openshift-ca\") pod \"91b85840-3a63-4c38-87ac-94d843f788d4\" (UID: \"91b85840-3a63-4c38-87ac-94d843f788d4\") "
Nov 02 15:28:38.717974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.717931    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-openshift-ca" pod "91b85840-3a63-4c38-87ac-94d843f788d4" (UID: "91b85840-3a63-4c38-87ac-94d843f788d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:38.719572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.719548    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b85840-3a63-4c38-87ac-94d843f788d4-kube-api-access-mwbqd" pod "91b85840-3a63-4c38-87ac-94d843f788d4" (UID: "91b85840-3a63-4c38-87ac-94d843f788d4"). InnerVolumeSpecName "kube-api-access-mwbqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:38.719769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.719673    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-xks-ca" pod "91b85840-3a63-4c38-87ac-94d843f788d4" (UID: "91b85840-3a63-4c38-87ac-94d843f788d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:28:38.792639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.792195    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"]
Nov 02 15:28:38.793970 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:28:38.793940    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde78ef6_9d08_402d_9fb4_5fa3a2b50074.slice/crio-0c0d3da29427aef16a326847f7b0c61431a61364801b971337dc97b4e45d235e WatchSource:0}: Error finding container 0c0d3da29427aef16a326847f7b0c61431a61364801b971337dc97b4e45d235e: Status 404 returned error can't find the container with id 0c0d3da29427aef16a326847f7b0c61431a61364801b971337dc97b4e45d235e
Nov 02 15:28:38.819391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.819356    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:38.819391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.819385    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbqd\" (UniqueName: \"kubernetes.io/projected/91b85840-3a63-4c38-87ac-94d843f788d4-kube-api-access-mwbqd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:38.819571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:38.819398    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b85840-3a63-4c38-87ac-94d843f788d4-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:39.241291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.241194    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq" event={"ID":"dde78ef6-9d08-402d-9fb4-5fa3a2b50074","Type":"ContainerStarted","Data":"7ff97cdf44b7ce87487fb1bd548f293fa14c56e73c1985b9e4885b7c8460cd78"}
Nov 02 15:28:39.241291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.241250    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq" event={"ID":"dde78ef6-9d08-402d-9fb4-5fa3a2b50074","Type":"ContainerStarted","Data":"0c0d3da29427aef16a326847f7b0c61431a61364801b971337dc97b4e45d235e"}
Nov 02 15:28:39.241802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.241402    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:39.242861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.242842    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"
Nov 02 15:28:39.242861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.242849    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4" event={"ID":"91b85840-3a63-4c38-87ac-94d843f788d4","Type":"ContainerDied","Data":"07258977fdbd253fd7f0b13189c9d3837e99376df3a9a3a5b9869e57f6e490a4"}
Nov 02 15:28:39.243065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.242898    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07258977fdbd253fd7f0b13189c9d3837e99376df3a9a3a5b9869e57f6e490a4"
Nov 02 15:28:39.244581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.244554    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" event={"ID":"d21df4f0-77ae-4f95-a5bc-0716effbdd3e","Type":"ContainerStarted","Data":"dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a"}
Nov 02 15:28:39.244712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.244606    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" event={"ID":"d21df4f0-77ae-4f95-a5bc-0716effbdd3e","Type":"ContainerStarted","Data":"0e64b631e338ea3fe2f616424fa248b203b3a13aa3181c194434d2bf6e0baec6"}
Nov 02 15:28:39.244712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.244686    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:39.246355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.246334    7019 generic.go:356] "Generic (PLEG): container finished" podID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerID="39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73" exitCode=0
Nov 02 15:28:39.246439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.246405    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" event={"ID":"d6d85dc4-dad6-4112-a6d0-ca4f29074444","Type":"ContainerDied","Data":"39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73"}
Nov 02 15:28:39.246815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.246797    7019 scope.go:122] "RemoveContainer" containerID="39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73"
Nov 02 15:28:39.248564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.248535    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:39.248632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.248605    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:39.248696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.248637    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:39.249997 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.249980    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" event={"ID":"d8db75dc-11e7-4eeb-9c6f-f3128863a73a","Type":"ContainerStarted","Data":"9fba8a6787ae0f3bfdaa8ed913a5dc0e7afaefe1f3f47c319f8697a8e25f48b0"}
Nov 02 15:28:39.286208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.286141    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq" podStartSLOduration=2.28612183 podStartE2EDuration="2.28612183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:39.283432064 +0000 UTC m=+1913.537385689" watchObservedRunningTime="2026-11-02 15:28:39.28612183 +0000 UTC m=+1913.540075455"
Nov 02 15:28:39.378510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.378444    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" podStartSLOduration=2.378425633 podStartE2EDuration="2.378425633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:39.373455438 +0000 UTC m=+1913.627409063" watchObservedRunningTime="2026-11-02 15:28:39.378425633 +0000 UTC m=+1913.632379259"
Nov 02 15:28:39.645702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.643408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:39.921712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.917830    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 15:28:39.921712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.917906    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 15:28:39.926446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:39.926409    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:28:40.005442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.005420    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:28:40.049167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049135    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b035673-1a9d-4150-956f-8c1512f0d857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b035673-1a9d-4150-956f-8c1512f0d857-istio-podinfo\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049284    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-socket\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049331    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-envoy\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049522    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-data\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049566    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-credential-socket\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049582    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-socket" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.049613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049598    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-istio-token\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-istio-token\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049726    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b035673-1a9d-4150-956f-8c1512f0d857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b035673-1a9d-4150-956f-8c1512f0d857-istiod-ca-cert\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049816    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-certs\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.049866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049817    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-data" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.049866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049834    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-credential-socket" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.050044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.049890    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-kube-api-access-n6px2\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-kube-api-access-n6px2\") pod \"3b035673-1a9d-4150-956f-8c1512f0d857\" (UID: \"3b035673-1a9d-4150-956f-8c1512f0d857\") "
Nov 02 15:28:40.050044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.050000    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-certs" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.050286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.050266    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b035673-1a9d-4150-956f-8c1512f0d857-istiod-ca-cert" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:28:40.050583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.050560    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.050678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.050587    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.050678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.050601    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b035673-1a9d-4150-956f-8c1512f0d857-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.050678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.050615    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.050678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.050628    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.052018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.051988    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b035673-1a9d-4150-956f-8c1512f0d857-istio-podinfo" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:28:40.052237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.052022    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-istio-token" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:40.052392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.052363    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-envoy" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:28:40.052514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.052442    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-kube-api-access-n6px2" pod "3b035673-1a9d-4150-956f-8c1512f0d857" (UID: "3b035673-1a9d-4150-956f-8c1512f0d857"). InnerVolumeSpecName "kube-api-access-n6px2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:28:40.151818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.151785    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b035673-1a9d-4150-956f-8c1512f0d857-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.151818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.151815    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.151818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.151825    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6px2\" (UniqueName: \"kubernetes.io/projected/3b035673-1a9d-4150-956f-8c1512f0d857-kube-api-access-n6px2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.152051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.151834    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b035673-1a9d-4150-956f-8c1512f0d857-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:28:40.260108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.260073    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" event={"ID":"d6d85dc4-dad6-4112-a6d0-ca4f29074444","Type":"ContainerStarted","Data":"96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4"}
Nov 02 15:28:40.261556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.261529    7019 generic.go:356] "Generic (PLEG): container finished" podID="3b035673-1a9d-4150-956f-8c1512f0d857" containerID="73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493" exitCode=0
Nov 02 15:28:40.261706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.261567    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"
Nov 02 15:28:40.261706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.261608    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" event={"ID":"3b035673-1a9d-4150-956f-8c1512f0d857","Type":"ContainerDied","Data":"73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493"}
Nov 02 15:28:40.261706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.261690    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk" event={"ID":"3b035673-1a9d-4150-956f-8c1512f0d857","Type":"ContainerDied","Data":"e4ac393d6728ea7b7068f3bd46d9f859f5112878b4e4f470fe9162c34f3bd2d3"}
Nov 02 15:28:40.261885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.261720    7019 scope.go:122] "RemoveContainer" containerID="73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493"
Nov 02 15:28:40.306873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.306846    7019 scope.go:122] "RemoveContainer" containerID="73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493"
Nov 02 15:28:40.307254 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:28:40.307226    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493\": container with ID starting with 73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493 not found: ID does not exist" containerID="73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493"
Nov 02 15:28:40.307372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.307263    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493"} err="failed to get container status \"73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493\": rpc error: code = NotFound desc = could not find container \"73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493\": container with ID starting with 73622eddbf1f5c332ca6a7cdc8e9451311bd5dcde1b75bb079055b8f88a40493 not found: ID does not exist"
Nov 02 15:28:40.359412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.359331    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"]
Nov 02 15:28:40.362854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.362828    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a713aa-openshift-default-7554947cc9-rsbdk"]
Nov 02 15:28:40.891348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.890047    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:28:40.902673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.894762    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:28:40.902673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.894822    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 15:28:40.915671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.908094    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 15:28:40.915671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:40.908169    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 15:28:41.433464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:41.433428    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:41.887309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:41.887234    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:28:41.887638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:41.887611    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 15:28:41.893678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:41.890887    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" path="/var/lib/kubelet/pods/3b035673-1a9d-4150-956f-8c1512f0d857/volumes"
Nov 02 15:28:41.902145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:41.902112    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 15:28:41.902389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:41.902358    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 15:28:42.886671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:42.886047    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:28:42.886671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:42.886126    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 15:28:42.908671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:42.905296    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 15:28:42.908671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:42.905371    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 15:28:43.061229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:43.061195    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:43.805683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:43.804770    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:28:43.888368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:43.886021    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:43.888368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:43.886628    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:28:43.888368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:43.886710    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 15:28:43.908303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:43.908155    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 15:28:43.908303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:43.908215    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 15:28:44.885670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:44.885057    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:28:44.885670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:44.885125    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 15:28:44.909681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:44.903074    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 15:28:44.909681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:44.903145    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 15:28:44.961708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:44.961666    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:45.042478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.041758    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:28:45.060221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.058768    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:28:45.556768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.556727    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"]
Nov 02 15:28:45.659111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.659067    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"]
Nov 02 15:28:45.660728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.660149    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy"
Nov 02 15:28:45.660728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.660170    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy"
Nov 02 15:28:45.660728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.660227    7019 cpu_manager.go:414] "containerMap: removing container" podUID="91b85840-3a63-4c38-87ac-94d843f788d4" containerName="revoke-keys"
Nov 02 15:28:45.660728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.660235    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b85840-3a63-4c38-87ac-94d843f788d4" containerName="revoke-keys"
Nov 02 15:28:45.660728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.660412    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b035673-1a9d-4150-956f-8c1512f0d857" containerName="istio-proxy"
Nov 02 15:28:45.660728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.660428    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b85840-3a63-4c38-87ac-94d843f788d4" containerName="revoke-keys"
Nov 02 15:28:45.669914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.669889    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.720465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.712108    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"default-dockercfg-rdkxk\""
Nov 02 15:28:45.720465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.712203    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"openshift-service-ca.crt\""
Nov 02 15:28:45.720465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.712326    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"test-model-2f8b7c-kserve-self-signed-certs\""
Nov 02 15:28:45.720465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.712390    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"kube-root-ca.crt\""
Nov 02 15:28:45.720465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.717507    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"]
Nov 02 15:28:45.762847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.755788    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-model-cache\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.762847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.755922    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4438617-bb51-496d-aee4-870ca57e9907-tls-certs\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.762847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.756007    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lpd\" (UniqueName: \"kubernetes.io/projected/b4438617-bb51-496d-aee4-870ca57e9907-kube-api-access-j4lpd\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.762847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.756083    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-dshm\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.762847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.756135    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-tmp-dir\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.762847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.756168    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-home\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.805689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.777095    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.867204    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-model-cache\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.867331    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4438617-bb51-496d-aee4-870ca57e9907-tls-certs\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.867414    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lpd\" (UniqueName: \"kubernetes.io/projected/b4438617-bb51-496d-aee4-870ca57e9907-kube-api-access-j4lpd\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.867514    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-dshm\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.867562    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-tmp-dir\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.867593    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-home\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.868173    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-home\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.868425    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-model-cache\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.876764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.869381    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-tmp-dir\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:45.943097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.898335    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:28:45.943097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.898393    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 15:28:45.943097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.898605    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"test-model-2f8b7c-kserve-self-signed-certs\""
Nov 02 15:28:45.943097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.916153    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"kube-root-ca.crt\""
Nov 02 15:28:45.943097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:45.942991    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"openshift-service-ca.crt\""
Nov 02 15:28:46.166456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.162532    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:46.254672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.254575    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4438617-bb51-496d-aee4-870ca57e9907-tls-certs\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:46.265687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.255199    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-dshm\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:46.336521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.336487    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:28:46.370800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.344947    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:28:46.393042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.392955    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:28:46.852719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.852277    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lpd\" (UniqueName: \"kubernetes.io/projected/b4438617-bb51-496d-aee4-870ca57e9907-kube-api-access-j4lpd\") pod \"test-model-2f8b7c-kserve-7c56cc97cf-96x4j\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") " pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:46.887720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.875109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:46.895481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.890266    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 15:28:46.900840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:46.900802    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 15:28:47.140996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.140962    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2f8b7c\"/\"default-dockercfg-rdkxk\""
Nov 02 15:28:47.145086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.145063    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:47.530530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.530437    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"]
Nov 02 15:28:47.626882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.625430    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j" event={"ID":"b4438617-bb51-496d-aee4-870ca57e9907","Type":"ContainerStarted","Data":"ef19d8b90c8e9aff28afb98ff12b1dd76c47e1372f8cc708a93f8f720691e0a2"}
Nov 02 15:28:47.724767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.720256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:47.906269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.906229    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:28:47.910197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.910168    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:28:48.000933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:47.996958    7019 scope.go:122] "RemoveContainer" containerID="b085c93cf4ee6df464b80d3a6620bf71dc6fd2fff737244a51da5e7ab3c42cdd"
Nov 02 15:28:48.023867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:48.022141    7019 scope.go:122] "RemoveContainer" containerID="7eb6cdf4f1ecbc09a72f03e7129080d5e4fc2b2eccdeff88a6d336670730caec"
Nov 02 15:28:48.049911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:48.045081    7019 scope.go:122] "RemoveContainer" containerID="8365b283cb66f45ca32e4a8331ff482920fdfde2572a38d8ddcef9197055ec2f"
Nov 02 15:28:48.605883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:48.605787    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:48.646825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:48.646746    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j" event={"ID":"b4438617-bb51-496d-aee4-870ca57e9907","Type":"ContainerStarted","Data":"bb178cd1e88a9ee045a71b41d7b001b0829828951addf4ab48889a5195c4cd17"}
Nov 02 15:28:48.696933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:48.696878    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j" podStartSLOduration=3.69686109 podStartE2EDuration="3.69686109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:28:48.692322918 +0000 UTC m=+1922.946276554" watchObservedRunningTime="2026-11-02 15:28:48.69686109 +0000 UTC m=+1922.950815039"
Nov 02 15:28:49.145884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:49.145847    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:50.292523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:50.291107    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:28:50.302679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:50.300849    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:28:50.318690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:50.318612    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:50.557924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:50.557821    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerName="payload-processing" containerID="cri-o://8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1" gracePeriod=25
Nov 02 15:28:50.722821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:50.722778    7019 patch_prober.go:29] interesting pod/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:28:50.722998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:50.722842    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:28:50.747854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:50.747821    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:28:51.521332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:51.521297    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:52.057606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:52.057563    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:52.679121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:52.679087    7019 generic.go:356] "Generic (PLEG): container finished" podID="ee941490-62a1-40fc-92f9-403df334d9bf" containerID="d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20" exitCode=0
Nov 02 15:28:52.680231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:52.679167    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" event={"ID":"ee941490-62a1-40fc-92f9-403df334d9bf","Type":"ContainerDied","Data":"d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20"}
Nov 02 15:28:52.680231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:52.679681    7019 scope.go:122] "RemoveContainer" containerID="d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20"
Nov 02 15:28:52.681736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:52.681705    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:52.681869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:52.681781    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:52.681869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:52.681820    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:28:52.711937 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:28:52.711905    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:28:53.705076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:53.705039    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" event={"ID":"ee941490-62a1-40fc-92f9-403df334d9bf","Type":"ContainerStarted","Data":"048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d"}
Nov 02 15:28:53.749139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:53.749104    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:28:53.749340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:53.749264    7019 patch_prober.go:29] interesting pod/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:28:53.749340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:53.749308    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 15:28:53.938315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:53.938280    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:54.749272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:54.749234    7019 patch_prober.go:29] interesting pod/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:28:54.749820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:54.749296    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 15:28:55.500206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:55.499733    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:55.749013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:55.748976    7019 patch_prober.go:29] interesting pod/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:28:55.749188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:55.749041    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 15:28:56.750160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:56.750126    7019 patch_prober.go:29] interesting pod/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:28:56.750638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:56.750189    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 15:28:57.064803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:57.060896    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:57.145795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:57.145757    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:57.159460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:57.159431    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:57.751156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:57.751117    7019 patch_prober.go:29] interesting pod/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:28:57.751781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:57.751747    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 15:28:57.798593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:57.798563    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:28:58.749835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:58.749801    7019 patch_prober.go:29] interesting pod/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body=
Nov 02 15:28:58.750015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:58.749870    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused"
Nov 02 15:28:58.797555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:58.797520    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:28:59.753919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:59.753888    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:28:59.754196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:59.754175    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:28:59.755038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:28:59.755015    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:29:00.350837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:00.350797    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:00.746768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:00.746726    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:29:01.480584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:01.480544    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:01.891699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:01.891631    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"]
Nov 02 15:29:01.896392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:01.896370    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:01.919221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:01.919187    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"]
Nov 02 15:29:01.937284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:01.936948    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-31c816-serving-cert\""
Nov 02 15:29:02.009246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.009205    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:02.044959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.044918    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495f0861-49cf-49fc-970e-f186501d48f8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.045151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.045130    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-maas-api-tls\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.045290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.045273    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-metrics-certs\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.045421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.045404    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtb5k\" (UniqueName: \"kubernetes.io/projected/495f0861-49cf-49fc-970e-f186501d48f8-kube-api-access-jtb5k\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.147282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.147194    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtb5k\" (UniqueName: \"kubernetes.io/projected/495f0861-49cf-49fc-970e-f186501d48f8-kube-api-access-jtb5k\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.147282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.147262    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495f0861-49cf-49fc-970e-f186501d48f8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.147563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.147409    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-maas-api-tls\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.147563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.147535    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-metrics-certs\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.148060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.148032    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495f0861-49cf-49fc-970e-f186501d48f8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.150062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.150028    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-maas-api-tls\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.150533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.150510    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-metrics-certs\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.174084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.174041    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtb5k\" (UniqueName: \"kubernetes.io/projected/495f0861-49cf-49fc-970e-f186501d48f8-kube-api-access-jtb5k\") pod \"maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.218964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.218927    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:02.226118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.226088    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:02.532371 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:29:02.532339    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495f0861_49cf_49fc_970e_f186501d48f8.slice/crio-df7623489cfa7da488709d00d4eb1c26b23599648cd34a46e3f866ee48a413e2 WatchSource:0}: Error finding container df7623489cfa7da488709d00d4eb1c26b23599648cd34a46e3f866ee48a413e2: Status 404 returned error can't find the container with id df7623489cfa7da488709d00d4eb1c26b23599648cd34a46e3f866ee48a413e2
Nov 02 15:29:02.534000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.533974    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"]
Nov 02 15:29:02.534166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.534147    7019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:29:02.808326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:02.808238    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7" event={"ID":"495f0861-49cf-49fc-970e-f186501d48f8","Type":"ContainerStarted","Data":"df7623489cfa7da488709d00d4eb1c26b23599648cd34a46e3f866ee48a413e2"}
Nov 02 15:29:03.787314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:03.787283    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:03.849991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:03.849952    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7" event={"ID":"495f0861-49cf-49fc-970e-f186501d48f8","Type":"ContainerStarted","Data":"1676d5af4b22179b5637df9eb5567d59810effc2ff9da82dec31aad8eafc58af"}
Nov 02 15:29:03.850168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:03.850127    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:03.900316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:03.900264    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7" podStartSLOduration=2.525245593 podStartE2EDuration="2.900249401s" totalImagesPullingTime="375.003808ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:03.89691124 +0000 UTC m=+1938.150864866" watchObservedRunningTime="2026-11-02 15:29:03.900249401 +0000 UTC m=+1938.154203025"
Nov 02 15:29:05.443885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:05.443846    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:07.007555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:07.007510    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:08.631873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:08.631841    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:09.865224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:09.865196    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:29:10.150076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.149989    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:10.746539 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:29:10.746511    7019 logging.go:55] [core] [Channel #1499 SubChannel #1500]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.111:50052", ServerName: "10.134.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.111:50052: connect: connection refused"
Nov 02 15:29:10.854691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.854659    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:29:10.912354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.912314    7019 generic.go:356] "Generic (PLEG): container finished" podID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerID="8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1" exitCode=0
Nov 02 15:29:10.912816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.912398    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" event={"ID":"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f","Type":"ContainerDied","Data":"8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1"}
Nov 02 15:29:10.912816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.912412    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"
Nov 02 15:29:10.912816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.912451    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" event={"ID":"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f","Type":"ContainerDied","Data":"a40aa1c015dfc4610561cd04bc79a3e7cf3d77ab51283b4fc807bb887e9e41da"}
Nov 02 15:29:10.912816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.912478    7019 scope.go:122] "RemoveContainer" containerID="8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1"
Nov 02 15:29:10.920138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.920113    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-kube-api-access-tftwh\" (UniqueName: \"kubernetes.io/projected/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-kube-api-access-tftwh\") pod \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") "
Nov 02 15:29:10.920343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.920322    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-plugins-config-volume\") pod \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\" (UID: \"85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f\") "
Nov 02 15:29:10.920903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.920876    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-plugins-config-volume" pod "85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" (UID: "85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:10.921406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.921379    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:10.923190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.923166    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-kube-api-access-tftwh" pod "85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" (UID: "85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f"). InnerVolumeSpecName "kube-api-access-tftwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:10.927510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.927488    7019 scope.go:122] "RemoveContainer" containerID="8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1"
Nov 02 15:29:10.928010 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:29:10.927981    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1\": container with ID starting with 8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1 not found: ID does not exist" containerID="8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1"
Nov 02 15:29:10.928109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.928021    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1"} err="failed to get container status \"8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1\": rpc error: code = NotFound desc = could not find container \"8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1\": container with ID starting with 8eaecd54340582ee43b089ca75018d34e95832457374bf9bc3a32ab79970a5e1 not found: ID does not exist"
Nov 02 15:29:10.941352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:10.941325    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:11.023447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:11.023407    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftwh\" (UniqueName: \"kubernetes.io/projected/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f-kube-api-access-tftwh\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:11.099035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:11.098946    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:11.268792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:11.268758    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"]
Nov 02 15:29:11.277075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:11.277042    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s"]
Nov 02 15:29:11.746144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:11.746098    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2bbcb5-7788b6d9f9-lq49s" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.111:50052\" within 1s: context deadline exceeded"
Nov 02 15:29:11.863719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:11.863682    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" path="/var/lib/kubelet/pods/85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f/volumes"
Nov 02 15:29:11.898319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:11.896806    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:13.492723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:13.492627    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:15.001708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:15.001671    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:16.543680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.543628    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:16.952551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.952516    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"]
Nov 02 15:29:16.953372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.953349    7019 cpu_manager.go:414] "containerMap: removing container" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerName="payload-processing"
Nov 02 15:29:16.953372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.953369    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerName="payload-processing"
Nov 02 15:29:16.953521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.953507    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c84f5f-3ab5-4cb9-b1d5-9b02d11aba0f" containerName="payload-processing"
Nov 02 15:29:16.958847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.958826    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:16.980136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.980107    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"]
Nov 02 15:29:16.986276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.986250    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d35a86\"/\"default-dockercfg-wpbrq\""
Nov 02 15:29:16.986466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.986416    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d35a86\"/\"test-model-d35a86-kserve-self-signed-certs\""
Nov 02 15:29:16.990424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.990400    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d35a86\"/\"kube-root-ca.crt\""
Nov 02 15:29:16.990628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:16.990614    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d35a86\"/\"openshift-service-ca.crt\""
Nov 02 15:29:17.019746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.019701    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-home\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.019943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.019886    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-tmp-dir\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.020017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.019953    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-dshm\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.020080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.020020    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-model-cache\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.020141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.020112    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e158e74-74d1-473c-a673-71119444b264-tls-certs\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.020197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.020151    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt7fx\" (UniqueName: \"kubernetes.io/projected/2e158e74-74d1-473c-a673-71119444b264-kube-api-access-xt7fx\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.050924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.050887    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:17.121242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121202    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-tmp-dir\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121280    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-dshm\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121349    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-model-cache\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121445    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e158e74-74d1-473c-a673-71119444b264-tls-certs\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121489    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt7fx\" (UniqueName: \"kubernetes.io/projected/2e158e74-74d1-473c-a673-71119444b264-kube-api-access-xt7fx\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121611    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-home\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121718    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-tmp-dir\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121784    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-model-cache\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.121963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.121938    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-home\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.123687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.123663    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-dshm\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.126072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.126051    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e158e74-74d1-473c-a673-71119444b264-tls-certs\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.150428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.150391    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt7fx\" (UniqueName: \"kubernetes.io/projected/2e158e74-74d1-473c-a673-71119444b264-kube-api-access-xt7fx\") pod \"test-model-d35a86-kserve-7988945bf-cmzbf\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") " pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.285146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.285050    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:17.729608 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:29:17.729571    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e158e74_74d1_473c_a673_71119444b264.slice/crio-273d4896a491cddf49946e8791af400882dd75b82cec8c32ba2ae6d1585a01fb WatchSource:0}: Error finding container 273d4896a491cddf49946e8791af400882dd75b82cec8c32ba2ae6d1585a01fb: Status 404 returned error can't find the container with id 273d4896a491cddf49946e8791af400882dd75b82cec8c32ba2ae6d1585a01fb
Nov 02 15:29:17.731117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.731087    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"]
Nov 02 15:29:17.991469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.991351    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf" event={"ID":"2e158e74-74d1-473c-a673-71119444b264","Type":"ContainerStarted","Data":"9a78dfbf474adc70dae8a8e167524c8fe6fc7755ba59b9e092871032c3d7f395"}
Nov 02 15:29:17.991469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:17.991404    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf" event={"ID":"2e158e74-74d1-473c-a673-71119444b264","Type":"ContainerStarted","Data":"273d4896a491cddf49946e8791af400882dd75b82cec8c32ba2ae6d1585a01fb"}
Nov 02 15:29:18.038289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:18.038234    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf" podStartSLOduration=2.038221545 podStartE2EDuration="2.038221545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:18.037697333 +0000 UTC m=+1952.291650957" watchObservedRunningTime="2026-11-02 15:29:18.038221545 +0000 UTC m=+1952.292175205"
Nov 02 15:29:18.285795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:18.285762    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:18.294837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:18.294796    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:19.769142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.769106    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:19.771124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.771090    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"]
Nov 02 15:29:19.777366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.777346    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.799450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.799418    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"]
Nov 02 15:29:19.820723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.820577    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28f621\"/\"openshift-service-ca.crt\""
Nov 02 15:29:19.820959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.820742    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28f621\"/\"kube-root-ca.crt\""
Nov 02 15:29:19.820959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.820870    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28f621\"/\"default-dockercfg-r97dg\""
Nov 02 15:29:19.821167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.820959    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28f621\"/\"test-model-28f621-kserve-self-signed-certs\""
Nov 02 15:29:19.875975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.875935    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh42s\" (UniqueName: \"kubernetes.io/projected/4a86c4cc-bc5e-4b27-b461-e5a590237005-kube-api-access-bh42s\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.876160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.876143    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86c4cc-bc5e-4b27-b461-e5a590237005-tls-certs\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.876302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.876276    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-home\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.876377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.876329    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-model-cache\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.876525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.876510    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-tmp-dir\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.876592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.876564    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-dshm\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.985673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.981867    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86c4cc-bc5e-4b27-b461-e5a590237005-tls-certs\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.985673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.982008    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-home\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.985673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.982055    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-model-cache\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.985673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.982225    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-tmp-dir\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.985673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.982279    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-dshm\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.985673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.982330    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh42s\" (UniqueName: \"kubernetes.io/projected/4a86c4cc-bc5e-4b27-b461-e5a590237005-kube-api-access-bh42s\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.989667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.988088    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-model-cache\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.989667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.988361    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-home\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:19.989667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.988605    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-tmp-dir\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:20.010691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:19.994632    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86c4cc-bc5e-4b27-b461-e5a590237005-tls-certs\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:20.020735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:20.016105    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-dshm\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:20.078392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:20.076942    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:20.113657 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:20.106230    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh42s\" (UniqueName: \"kubernetes.io/projected/4a86c4cc-bc5e-4b27-b461-e5a590237005-kube-api-access-bh42s\") pod \"test-model-28f621-kserve-69984cddd9-zmtlh\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") " pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:20.141670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:20.123956    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:20.490893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:20.489853    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"]
Nov 02 15:29:20.490893 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:29:20.489995    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a86c4cc_bc5e_4b27_b461_e5a590237005.slice/crio-5dac7f2a87180d1feec7d82ebfb58740ed68158f85dceff57eedb7e3ad4e718d WatchSource:0}: Error finding container 5dac7f2a87180d1feec7d82ebfb58740ed68158f85dceff57eedb7e3ad4e718d: Status 404 returned error can't find the container with id 5dac7f2a87180d1feec7d82ebfb58740ed68158f85dceff57eedb7e3ad4e718d
Nov 02 15:29:21.124387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:21.124349    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh" event={"ID":"4a86c4cc-bc5e-4b27-b461-e5a590237005","Type":"ContainerStarted","Data":"8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4"}
Nov 02 15:29:21.124859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:21.124399    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh" event={"ID":"4a86c4cc-bc5e-4b27-b461-e5a590237005","Type":"ContainerStarted","Data":"5dac7f2a87180d1feec7d82ebfb58740ed68158f85dceff57eedb7e3ad4e718d"}
Nov 02 15:29:21.124859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:21.124441    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:21.174243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:21.174190    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh" podStartSLOduration=2.174175711 podStartE2EDuration="2.174175711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:21.170232854 +0000 UTC m=+1955.424186482" watchObservedRunningTime="2026-11-02 15:29:21.174175711 +0000 UTC m=+1955.428129716"
Nov 02 15:29:21.553478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:21.553374    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:22.709667 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:29:22.709175    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:29:23.109031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:23.108940    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:23.248449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:23.248412    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:23.545128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:23.545095    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:24.723497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.723468    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"]
Nov 02 15:29:24.728715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.728687    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.747700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.747671    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"]
Nov 02 15:29:24.757352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.757324    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cfc63a\"/\"kube-root-ca.crt\""
Nov 02 15:29:24.757945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.757922    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cfc63a\"/\"openshift-service-ca.crt\""
Nov 02 15:29:24.758066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.758033    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cfc63a\"/\"default-dockercfg-knzvn\""
Nov 02 15:29:24.759769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.759748    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cfc63a\"/\"test-model-cfc63a-kserve-self-signed-certs\""
Nov 02 15:29:24.791960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.779625    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcn4c\" (UniqueName: \"kubernetes.io/projected/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-kube-api-access-vcn4c\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.791960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.779946    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tls-certs\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.791960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.779995    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tmp-dir\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.791960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.780031    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-home\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.791960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.780074    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-dshm\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.791960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.780097    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-model-cache\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.812885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.812849    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:24.881603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.881553    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tls-certs\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.881817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.881633    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tmp-dir\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.881817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.881693    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-home\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.881817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.881749    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-dshm\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.881817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.881771    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-model-cache\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.882076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.882037    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcn4c\" (UniqueName: \"kubernetes.io/projected/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-kube-api-access-vcn4c\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.882334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.882308    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-model-cache\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.882584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.882562    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tmp-dir\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.883048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.883022    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-home\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.886718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.886688    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-dshm\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.893955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.893929    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tls-certs\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.908926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.908893    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcn4c\" (UniqueName: \"kubernetes.io/projected/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-kube-api-access-vcn4c\") pod \"test-model-cfc63a-kserve-656bf444dc-x4t68\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") " pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:24.952423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:24.952388    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:25.056777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:25.056083    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:25.487130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:25.487099    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"]
Nov 02 15:29:25.487258 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:29:25.487155    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08f14532_73ef_4bbd_9d31_2e2d0700f0b6.slice/crio-5040806dd3c8bfa1e704160597129e24d3c9d00e821e002862008f76209ce135 WatchSource:0}: Error finding container 5040806dd3c8bfa1e704160597129e24d3c9d00e821e002862008f76209ce135: Status 404 returned error can't find the container with id 5040806dd3c8bfa1e704160597129e24d3c9d00e821e002862008f76209ce135
Nov 02 15:29:26.216508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:26.216460    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68" event={"ID":"08f14532-73ef-4bbd-9d31-2e2d0700f0b6","Type":"ContainerStarted","Data":"235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf"}
Nov 02 15:29:26.216508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:26.216513    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68" event={"ID":"08f14532-73ef-4bbd-9d31-2e2d0700f0b6","Type":"ContainerStarted","Data":"5040806dd3c8bfa1e704160597129e24d3c9d00e821e002862008f76209ce135"}
Nov 02 15:29:26.264032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:26.263981    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68" podStartSLOduration=2.263966659 podStartE2EDuration="2.263966659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:29:26.261331755 +0000 UTC m=+1960.515285379" watchObservedRunningTime="2026-11-02 15:29:26.263966659 +0000 UTC m=+1960.517920282"
Nov 02 15:29:26.405694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:26.405629    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:27.056860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:27.056831    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:27.285838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:27.285805    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:27.299765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:27.299736    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:27.901691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:27.901620    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:28.036759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:28.036721    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:28.245298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:28.245212    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:29:29.493038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:29.493002    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:29.972675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:29.955667    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:29:30.125298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:30.125264    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:30.138935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:30.138899    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:30.261230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:30.261148    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:29:31.066684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:31.059350    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:32.060495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.060458    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:29:32.062734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.062692    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 15:29:32.063506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.063475    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:32.125824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.125764    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"]
Nov 02 15:29:32.142245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.142196    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2bbcb5-84hw4"]
Nov 02 15:29:32.143630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.143211    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 02 15:29:32.171926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.168997    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq" podUID="816f42c0-98fd-409c-8a8c-52350a1ba4c1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.135:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:29:32.268672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.265078    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:32.535871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:32.535836    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:33.863289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:33.863254    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b85840-3a63-4c38-87ac-94d843f788d4" path="/var/lib/kubelet/pods/91b85840-3a63-4c38-87ac-94d843f788d4/volumes"
Nov 02 15:29:33.922854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:33.922813    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:35.057201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:35.056575    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:35.123258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:35.123000    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:35.419474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:35.419422    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:29:35.602603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:35.602569    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:37.203468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:37.203427    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:38.886600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:38.886566    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:39.047874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:39.047841    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:39.222776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:39.222587    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"]
Nov 02 15:29:39.222965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:39.222822    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" containerID="cri-o://918f385c146a1bfcb370062b8d8889cd90d8ca89d9d5767dea7c0445f657d405" gracePeriod=30
Nov 02 15:29:39.225309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:39.225155    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:39.365676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:39.364269    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:29:39.365676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:39.364342    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:29:40.082440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:40.082407    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:40.601445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:40.601410    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:40.760674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:40.755278    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:40.904163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:40.904128    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:42.382663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.382624    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:42.544258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.544159    7019 generic.go:356] "Generic (PLEG): container finished" podID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerID="89deddf99ecfba4fdcef094eff85129c2a1abd5d4be6a7f6b41c010a987b1d83" exitCode=0
Nov 02 15:29:42.544258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.544193    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" event={"ID":"ab9b3c83-b754-4bbe-94fa-8036234b0e65","Type":"ContainerDied","Data":"89deddf99ecfba4fdcef094eff85129c2a1abd5d4be6a7f6b41c010a987b1d83"}
Nov 02 15:29:42.544625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.544606    7019 scope.go:122] "RemoveContainer" containerID="89deddf99ecfba4fdcef094eff85129c2a1abd5d4be6a7f6b41c010a987b1d83"
Nov 02 15:29:42.546589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.546538    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:29:42.546714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.546643    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:29:42.546785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.546735    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:29:42.568188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:42.565631    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:29:43.554854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:43.554814    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" event={"ID":"ab9b3c83-b754-4bbe-94fa-8036234b0e65","Type":"ContainerStarted","Data":"dda6dfc4286d63a17ef4bc7a517faa0ff1991ea43f215922b4abefd95e011287"}
Nov 02 15:29:44.196672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.192221    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:44.351720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.346306    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:44.366293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.363672    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:29:44.366405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.366368    7019 patch_prober.go:29] interesting pod/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:29:44.366475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.366433    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 15:29:44.651459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.650958    7019 generic.go:356] "Generic (PLEG): container finished" podID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerID="918f385c146a1bfcb370062b8d8889cd90d8ca89d9d5767dea7c0445f657d405" exitCode=0
Nov 02 15:29:44.651459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.651113    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" event={"ID":"62dde604-430f-4f5f-bae4-111c194a1d2b","Type":"ContainerDied","Data":"918f385c146a1bfcb370062b8d8889cd90d8ca89d9d5767dea7c0445f657d405"}
Nov 02 15:29:44.651459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.651203    7019 scope.go:122] "RemoveContainer" containerID="fddb1e4877332ad6835a7f515ef91326f53adf852fc83bb763a55975c5bc6077"
Nov 02 15:29:44.814072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.814040    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:29:44.954058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.954022    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-data\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.954245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.954119    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-podinfo\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.954245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.954228    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-token\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-token\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.954376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.954345    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-data" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.954686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.954628    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-envoy\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.954998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.954949    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-certs\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.955261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.955241    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-credential-socket\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.955677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.955275    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-certs" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.955777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.955580    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-credential-socket" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.955925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.955906    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62dde604-430f-4f5f-bae4-111c194a1d2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dde604-430f-4f5f-bae4-111c194a1d2b-istiod-ca-cert\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.956476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.956360    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62dde604-430f-4f5f-bae4-111c194a1d2b-istiod-ca-cert" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:29:44.956578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.956542    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-kube-api-access-gd66q\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-kube-api-access-gd66q\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.956668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.956622    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-socket\") pod \"62dde604-430f-4f5f-bae4-111c194a1d2b\" (UID: \"62dde604-430f-4f5f-bae4-111c194a1d2b\") "
Nov 02 15:29:44.957195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.957157    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-socket" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.957447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.957357    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-podinfo" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:29:44.957447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.957365    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-envoy" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:29:44.957447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.957415    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-token" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:44.958281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.958260    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.958413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.958399    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.958517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.958505    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.958611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.958599    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.958793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.958778    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.958902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.958889    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.958996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.958983    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62dde604-430f-4f5f-bae4-111c194a1d2b-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.959102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.959081    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62dde604-430f-4f5f-bae4-111c194a1d2b-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:44.960099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:44.960070    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-kube-api-access-gd66q" pod "62dde604-430f-4f5f-bae4-111c194a1d2b" (UID: "62dde604-430f-4f5f-bae4-111c194a1d2b"). InnerVolumeSpecName "kube-api-access-gd66q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:29:45.060768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.060713    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd66q\" (UniqueName: \"kubernetes.io/projected/62dde604-430f-4f5f-bae4-111c194a1d2b-kube-api-access-gd66q\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:29:45.358133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.358050    7019 patch_prober.go:29] interesting pod/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:29:45.358133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.358105    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 15:29:45.660314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.660283    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"
Nov 02 15:29:45.660771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.660337    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd" event={"ID":"62dde604-430f-4f5f-bae4-111c194a1d2b","Type":"ContainerDied","Data":"abc557bbbc159a398f6e1380816a0b999a40618535b6213290c5c78ddbb3bfdf"}
Nov 02 15:29:45.660771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.660395    7019 scope.go:122] "RemoveContainer" containerID="918f385c146a1bfcb370062b8d8889cd90d8ca89d9d5767dea7c0445f657d405"
Nov 02 15:29:45.749567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.749541    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"]
Nov 02 15:29:45.753602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.753575    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2bbcb5-openshift-default-788c5c5b65-bvjtd"]
Nov 02 15:29:45.905673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.904164    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" path="/var/lib/kubelet/pods/62dde604-430f-4f5f-bae4-111c194a1d2b/volumes"
Nov 02 15:29:45.999379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:45.999267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:46.159694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:46.159640    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:46.332403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:46.332307    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:46.428256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:46.428219    7019 patch_prober.go:29] interesting pod/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:29:46.428530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:46.428500    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 15:29:47.358236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:47.358205    7019 patch_prober.go:29] interesting pod/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:29:47.358668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:47.358253    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 15:29:47.829413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:47.829378    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:47.998819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:47.998784    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:48.122615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:48.122582    7019 scope.go:122] "RemoveContainer" containerID="2d19a6a0f057dcec59309baf798bfd882a669a724a3b6c8ae31c4ee5599f140b"
Nov 02 15:29:48.133104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:48.133081    7019 scope.go:122] "RemoveContainer" containerID="9e369823aa7178e38b66b4e83bf01f1dba0e00d09a6f83aa356b2602ef646375"
Nov 02 15:29:48.358119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:48.358087    7019 patch_prober.go:29] interesting pod/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:29:48.358316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:48.358147    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 15:29:49.357936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:49.357896    7019 patch_prober.go:29] interesting pod/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 02 15:29:49.358145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:49.357967    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 02 15:29:49.485322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:49.485286    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:50.363141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:50.363102    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:29:50.363548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:50.363524    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:29:50.364217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:50.364199    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:29:51.153422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:51.153388    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:52.707418 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:29:52.707378    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:29:52.814243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:52.814208    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:54.435469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:54.435425    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:56.087761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:56.087715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:56.386094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:56.386056    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:57.810956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:57.810916    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:59.191562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.191520    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"]
Nov 02 15:29:59.192739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.192714    7019 cpu_manager.go:414] "containerMap: removing container" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy"
Nov 02 15:29:59.192739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.192737    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy"
Nov 02 15:29:59.192889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.192760    7019 cpu_manager.go:414] "containerMap: removing container" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy"
Nov 02 15:29:59.192889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.192768    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy"
Nov 02 15:29:59.192956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.192948    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy"
Nov 02 15:29:59.196815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.196796    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.223424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.223387    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"]
Nov 02 15:29:59.227377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.227263    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80ec03\"/\"kube-root-ca.crt\""
Nov 02 15:29:59.227377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.227308    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80ec03\"/\"default-dockercfg-2gr9p\""
Nov 02 15:29:59.227377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.227377    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80ec03\"/\"openshift-service-ca.crt\""
Nov 02 15:29:59.227589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.227395    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80ec03\"/\"test-model-80ec03-kserve-self-signed-certs\""
Nov 02 15:29:59.286444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.286410    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-dshm\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.286624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.286478    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-tmp-dir\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.286624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.286575    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nb79\" (UniqueName: \"kubernetes.io/projected/9888de5c-77fa-4a4a-b931-11e46a4946a6-kube-api-access-9nb79\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.286737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.286623    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9888de5c-77fa-4a4a-b931-11e46a4946a6-tls-certs\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.286789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.286763    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-home\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.286840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.286805    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-model-cache\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.306667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.306623    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:59.388245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.388203    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-model-cache\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.388482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.388459    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-dshm\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.388592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.388573    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-tmp-dir\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.388711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.388660    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nb79\" (UniqueName: \"kubernetes.io/projected/9888de5c-77fa-4a4a-b931-11e46a4946a6-kube-api-access-9nb79\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.388711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.388673    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-model-cache\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.388845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.388731    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9888de5c-77fa-4a4a-b931-11e46a4946a6-tls-certs\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.389055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.389024    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-home\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.389432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.389404    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-home\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.389635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.389610    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-tmp-dir\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.392259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.392215    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-dshm\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.396126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.396090    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9888de5c-77fa-4a4a-b931-11e46a4946a6-tls-certs\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.417091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.417062    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nb79\" (UniqueName: \"kubernetes.io/projected/9888de5c-77fa-4a4a-b931-11e46a4946a6-kube-api-access-9nb79\") pod \"test-model-80ec03-kserve-994f47f6f-xrk7t\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") " pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.455192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.455106    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:29:59.526966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.526931    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:29:59.963314 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:29:59.963248    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9888de5c_77fa_4a4a_b931_11e46a4946a6.slice/crio-9ace56f3618243239248ab2f102bfc3b61bc2f37d477e61a1af4292a2c1e1cad WatchSource:0}: Error finding container 9ace56f3618243239248ab2f102bfc3b61bc2f37d477e61a1af4292a2c1e1cad: Status 404 returned error can't find the container with id 9ace56f3618243239248ab2f102bfc3b61bc2f37d477e61a1af4292a2c1e1cad
Nov 02 15:29:59.964535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:29:59.964510    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"]
Nov 02 15:30:00.168992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.168839    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"]
Nov 02 15:30:00.172018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.171985    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="62dde604-430f-4f5f-bae4-111c194a1d2b" containerName="istio-proxy"
Nov 02 15:30:00.177796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.177473    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.179640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.179613    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"]
Nov 02 15:30:00.188038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.185948    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"]
Nov 02 15:30:00.193193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.191773    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.200147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.200111    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"]
Nov 02 15:30:00.211722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.202176    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.211722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.202879    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p79dv\" (UniqueName: \"kubernetes.io/projected/1eb97b1a-5e61-4668-a9c4-31769504ed43-kube-api-access-p79dv\") pod \"maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.211722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.203024    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb97b1a-5e61-4668-a9c4-31769504ed43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.211722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.203179    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdl68\" (UniqueName: \"kubernetes.io/projected/70b86377-400c-4ec3-8444-5e5c62e01fae-kube-api-access-wdl68\") pod \"maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.211722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.203222    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b86377-400c-4ec3-8444-5e5c62e01fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.211722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.208625    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:30:00.212445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.212425    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.213781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.213717    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"]
Nov 02 15:30:00.219876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.219859    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.231897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.228257    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"]
Nov 02 15:30:00.254618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.253626    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"]
Nov 02 15:30:00.265663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.264624    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"]
Nov 02 15:30:00.285838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.285807    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"]
Nov 02 15:30:00.295591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.294365    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"]
Nov 02 15:30:00.299759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.299607    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"]
Nov 02 15:30:00.300281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.299715    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.304726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.304701    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.304918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.304899    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdl68\" (UniqueName: \"kubernetes.io/projected/70b86377-400c-4ec3-8444-5e5c62e01fae-kube-api-access-wdl68\") pod \"maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.305044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.305027    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5748e924-6bf8-41b2-b5f6-b9e681456dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.305176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.305161    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b86377-400c-4ec3-8444-5e5c62e01fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.305291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.305276    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlmj\" (UniqueName: \"kubernetes.io/projected/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-kube-api-access-4tlmj\") pod \"maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.308069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.305498    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.308818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.308785    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p79dv\" (UniqueName: \"kubernetes.io/projected/1eb97b1a-5e61-4668-a9c4-31769504ed43-kube-api-access-p79dv\") pod \"maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.309134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.309112    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmw2n\" (UniqueName: \"kubernetes.io/projected/5748e924-6bf8-41b2-b5f6-b9e681456dcb-kube-api-access-qmw2n\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.309344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.309309    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb97b1a-5e61-4668-a9c4-31769504ed43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.309528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.309501    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqj7j\" (UniqueName: \"kubernetes.io/projected/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-kube-api-access-pqj7j\") pod \"maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.311262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.311241    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b86377-400c-4ec3-8444-5e5c62e01fae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.315001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.313001    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb97b1a-5e61-4668-a9c4-31769504ed43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.369915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.369883    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"]
Nov 02 15:30:00.382806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.382780    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.438405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.438367    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlmj\" (UniqueName: \"kubernetes.io/projected/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-kube-api-access-4tlmj\") pod \"maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.438737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.438709    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwj8k\" (UniqueName: \"kubernetes.io/projected/55911434-6f57-4840-a09a-a67befd78f30-kube-api-access-wwj8k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.438977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.438957    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.439140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.439120    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.439356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.439337    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55911434-6f57-4840-a09a-a67befd78f30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.439852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.439828    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmw2n\" (UniqueName: \"kubernetes.io/projected/5748e924-6bf8-41b2-b5f6-b9e681456dcb-kube-api-access-qmw2n\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.440881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.440858    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njm9t\" (UniqueName: \"kubernetes.io/projected/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-kube-api-access-njm9t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.441069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.441050    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqj7j\" (UniqueName: \"kubernetes.io/projected/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-kube-api-access-pqj7j\") pod \"maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.441290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.441272    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.441435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.441418    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5748e924-6bf8-41b2-b5f6-b9e681456dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.442472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.442449    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5748e924-6bf8-41b2-b5f6-b9e681456dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.453260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.448679    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.453368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.448801    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.456536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.456507    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p79dv\" (UniqueName: \"kubernetes.io/projected/1eb97b1a-5e61-4668-a9c4-31769504ed43-kube-api-access-p79dv\") pod \"maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.459476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.459447    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"]
Nov 02 15:30:00.459593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.459500    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"]
Nov 02 15:30:00.459593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.459525    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"]
Nov 02 15:30:00.459593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.459551    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"]
Nov 02 15:30:00.459907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.459891    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.502716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.502027    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdl68\" (UniqueName: \"kubernetes.io/projected/70b86377-400c-4ec3-8444-5e5c62e01fae-kube-api-access-wdl68\") pod \"maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.505673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.504993    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"]
Nov 02 15:30:00.507107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.507049    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlmj\" (UniqueName: \"kubernetes.io/projected/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-kube-api-access-4tlmj\") pod \"maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.509299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.509275    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.513116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.513089    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:00.527698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.524861    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"]
Nov 02 15:30:00.527698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.524931    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"]
Nov 02 15:30:00.545781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.541286    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.550182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550113    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55911434-6f57-4840-a09a-a67befd78f30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.550312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550180    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a57c32-b7d5-440a-adbc-6161142b71db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.550312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550304    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.550442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550353    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njm9t\" (UniqueName: \"kubernetes.io/projected/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-kube-api-access-njm9t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.550442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550394    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c84259f-d067-478d-ac0d-4c6087b95a2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.550567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550476    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qwb\" (UniqueName: \"kubernetes.io/projected/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-kube-api-access-s2qwb\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.550567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550536    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzbc\" (UniqueName: \"kubernetes.io/projected/a2a57c32-b7d5-440a-adbc-6161142b71db-kube-api-access-jhzbc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.550768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550631    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/3c84259f-d067-478d-ac0d-4c6087b95a2d-kube-api-access-2bgkr\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.550768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550731    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwj8k\" (UniqueName: \"kubernetes.io/projected/55911434-6f57-4840-a09a-a67befd78f30-kube-api-access-wwj8k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.550924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.550892    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.551749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.551509    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:00.552302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.552270    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.553095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.553070    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55911434-6f57-4840-a09a-a67befd78f30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.555204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.555180    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"]
Nov 02 15:30:00.556716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.556639    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.569722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.568388    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"]
Nov 02 15:30:00.569722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.568443    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"]
Nov 02 15:30:00.569722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.568560    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.578839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.578807    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njm9t\" (UniqueName: \"kubernetes.io/projected/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-kube-api-access-njm9t\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.584670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.581587    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"]
Nov 02 15:30:00.584670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.584255    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwj8k\" (UniqueName: \"kubernetes.io/projected/55911434-6f57-4840-a09a-a67befd78f30-kube-api-access-wwj8k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.593885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.592306    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmw2n\" (UniqueName: \"kubernetes.io/projected/5748e924-6bf8-41b2-b5f6-b9e681456dcb-kube-api-access-qmw2n\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.597827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.597801    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"]
Nov 02 15:30:00.598394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.598370    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:00.600162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.600135    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqj7j\" (UniqueName: \"kubernetes.io/projected/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-kube-api-access-pqj7j\") pod \"maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.602247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.601837    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:00.611490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.611469    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"]
Nov 02 15:30:00.611607 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.611572    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:00.619406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.618866    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"]
Nov 02 15:30:00.624513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.624438    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"]
Nov 02 15:30:00.628339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.628158    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:00.630586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.630568    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"]
Nov 02 15:30:00.660484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660456    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.660629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660507    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ca184a-7c3f-43fb-acb6-dce5d342b804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.660629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660540    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffn4l\" (UniqueName: \"kubernetes.io/projected/63ca184a-7c3f-43fb-acb6-dce5d342b804-kube-api-access-ffn4l\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.660629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660575    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c84259f-d067-478d-ac0d-4c6087b95a2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.660863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660640    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qwb\" (UniqueName: \"kubernetes.io/projected/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-kube-api-access-s2qwb\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.660863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660698    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzbc\" (UniqueName: \"kubernetes.io/projected/a2a57c32-b7d5-440a-adbc-6161142b71db-kube-api-access-jhzbc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.660863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660774    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/3c84259f-d067-478d-ac0d-4c6087b95a2d-kube-api-access-2bgkr\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.660863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660807    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a739bb4-d8f5-4b46-a635-d486b4d7e314-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.660863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660842    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c69419-8ae3-488b-ae72-137540b1afc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:00.661145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.660997    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w62lc\" (UniqueName: \"kubernetes.io/projected/4a739bb4-d8f5-4b46-a635-d486b4d7e314-kube-api-access-w62lc\") pod \"maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.661145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.661080    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdjn\" (UniqueName: \"kubernetes.io/projected/91c69419-8ae3-488b-ae72-137540b1afc8-kube-api-access-frdjn\") pod \"maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:00.661256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.661151    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a57c32-b7d5-440a-adbc-6161142b71db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.661256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.661203    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bp4p\" (UniqueName: \"kubernetes.io/projected/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-kube-api-access-9bp4p\") pod \"maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:00.661256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.661239    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:00.664166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.662737    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c84259f-d067-478d-ac0d-4c6087b95a2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.664166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.663160    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.664166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.664125    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a57c32-b7d5-440a-adbc-6161142b71db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.672260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.671170    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:00.716234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.715330    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/3c84259f-d067-478d-ac0d-4c6087b95a2d-kube-api-access-2bgkr\") pod \"maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.716234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.715940    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qwb\" (UniqueName: \"kubernetes.io/projected/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-kube-api-access-s2qwb\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.717978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.717934    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzbc\" (UniqueName: \"kubernetes.io/projected/a2a57c32-b7d5-440a-adbc-6161142b71db-kube-api-access-jhzbc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763272    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bp4p\" (UniqueName: \"kubernetes.io/projected/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-kube-api-access-9bp4p\") pod \"maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763325    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763422    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ca184a-7c3f-43fb-acb6-dce5d342b804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763453    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffn4l\" (UniqueName: \"kubernetes.io/projected/63ca184a-7c3f-43fb-acb6-dce5d342b804-kube-api-access-ffn4l\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763633    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a739bb4-d8f5-4b46-a635-d486b4d7e314-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763683    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c69419-8ae3-488b-ae72-137540b1afc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763834    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w62lc\" (UniqueName: \"kubernetes.io/projected/4a739bb4-d8f5-4b46-a635-d486b4d7e314-kube-api-access-w62lc\") pod \"maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.765119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.763916    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdjn\" (UniqueName: \"kubernetes.io/projected/91c69419-8ae3-488b-ae72-137540b1afc8-kube-api-access-frdjn\") pod \"maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:00.769455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.766918    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:00.769455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.767639    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ca184a-7c3f-43fb-acb6-dce5d342b804-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.769455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.768547    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a739bb4-d8f5-4b46-a635-d486b4d7e314-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.769455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.769182    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c69419-8ae3-488b-ae72-137540b1afc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:00.807929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.807706    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:00.815472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.815167    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdjn\" (UniqueName: \"kubernetes.io/projected/91c69419-8ae3-488b-ae72-137540b1afc8-kube-api-access-frdjn\") pod \"maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:00.817552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.817477    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w62lc\" (UniqueName: \"kubernetes.io/projected/4a739bb4-d8f5-4b46-a635-d486b4d7e314-kube-api-access-w62lc\") pod \"maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.822272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.821913    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bp4p\" (UniqueName: \"kubernetes.io/projected/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-kube-api-access-9bp4p\") pod \"maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:00.825753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.825247    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffn4l\" (UniqueName: \"kubernetes.io/projected/63ca184a-7c3f-43fb-acb6-dce5d342b804-kube-api-access-ffn4l\") pod \"maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.867677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.867421    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t" event={"ID":"9888de5c-77fa-4a4a-b931-11e46a4946a6","Type":"ContainerStarted","Data":"d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d"}
Nov 02 15:30:00.867677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.867480    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t" event={"ID":"9888de5c-77fa-4a4a-b931-11e46a4946a6","Type":"ContainerStarted","Data":"9ace56f3618243239248ab2f102bfc3b61bc2f37d477e61a1af4292a2c1e1cad"}
Nov 02 15:30:00.873134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.873108    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:00.889951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.889922    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:00.922512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.918491    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:00.926771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.926240    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:00.948307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.948081    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:00.963026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.960217    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:00.988931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:00.986770    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"]
Nov 02 15:30:00.993264 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:30:00.993215    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b86377_400c_4ec3_8444_5e5c62e01fae.slice/crio-c2de8317256a7c9d45c1ef1af733b58ffeb6e10d645372e009ad68ed87d68d13 WatchSource:0}: Error finding container c2de8317256a7c9d45c1ef1af733b58ffeb6e10d645372e009ad68ed87d68d13: Status 404 returned error can't find the container with id c2de8317256a7c9d45c1ef1af733b58ffeb6e10d645372e009ad68ed87d68d13
Nov 02 15:30:01.013677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.013458    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:01.037560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.037256    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:01.044286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.039339    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t" podStartSLOduration=2.039321891 podStartE2EDuration="2.039321891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:01.029385854 +0000 UTC m=+1995.283339482" watchObservedRunningTime="2026-11-02 15:30:01.039321891 +0000 UTC m=+1995.293275516"
Nov 02 15:30:01.264890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.263611    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:01.308089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.293161    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"]
Nov 02 15:30:01.308089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.299792    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"]
Nov 02 15:30:01.370761 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:30:01.329263    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb97b1a_5e61_4668_a9c4_31769504ed43.slice/crio-2120521ac44c1eef68309ad0b1a30b0b6d44a2af6cde5afcdd76f3bc2164b7f1 WatchSource:0}: Error finding container 2120521ac44c1eef68309ad0b1a30b0b6d44a2af6cde5afcdd76f3bc2164b7f1: Status 404 returned error can't find the container with id 2120521ac44c1eef68309ad0b1a30b0b6d44a2af6cde5afcdd76f3bc2164b7f1
Nov 02 15:30:01.527338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.527302    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:30:01.630786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.629841    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"]
Nov 02 15:30:01.646820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.646775    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:01.664949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.664920    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"]
Nov 02 15:30:01.680018 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:30:01.679129    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55911434_6f57_4840_a09a_a67befd78f30.slice/crio-57a6735b683834a2a7355773df6e3bb992651ef916b7aef317751a8e41dd1df5 WatchSource:0}: Error finding container 57a6735b683834a2a7355773df6e3bb992651ef916b7aef317751a8e41dd1df5: Status 404 returned error can't find the container with id 57a6735b683834a2a7355773df6e3bb992651ef916b7aef317751a8e41dd1df5
Nov 02 15:30:01.726669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.725807    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"]
Nov 02 15:30:01.726669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.725879    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"]
Nov 02 15:30:01.740830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.732223    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"]
Nov 02 15:30:01.745724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.745697    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"]
Nov 02 15:30:01.757703 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:30:01.756165    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5748e924_6bf8_41b2_b5f6_b9e681456dcb.slice/crio-7a367ed94ebc96d9cf3b4838f3ade0fc15715c688daf993c6747225c7d49a5a8 WatchSource:0}: Error finding container 7a367ed94ebc96d9cf3b4838f3ade0fc15715c688daf993c6747225c7d49a5a8: Status 404 returned error can't find the container with id 7a367ed94ebc96d9cf3b4838f3ade0fc15715c688daf993c6747225c7d49a5a8
Nov 02 15:30:01.809881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.809844    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"]
Nov 02 15:30:01.819936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.817084    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"]
Nov 02 15:30:01.824442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.824414    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"]
Nov 02 15:30:01.888494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.888469    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" event={"ID":"3c84259f-d067-478d-ac0d-4c6087b95a2d","Type":"ContainerStarted","Data":"8e083084bdf98217f078a5de769129ff59df51df6cf20f09d413ba3de2537988"}
Nov 02 15:30:01.890000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.889972    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" event={"ID":"5748e924-6bf8-41b2-b5f6-b9e681456dcb","Type":"ContainerStarted","Data":"7a367ed94ebc96d9cf3b4838f3ade0fc15715c688daf993c6747225c7d49a5a8"}
Nov 02 15:30:01.892995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.892806    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" event={"ID":"91c69419-8ae3-488b-ae72-137540b1afc8","Type":"ContainerStarted","Data":"bdb7458801dbd7b9a9d559b2172e38327447bed7c6fd8d999abf7b6140eb11d5"}
Nov 02 15:30:01.895914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.895885    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" event={"ID":"1eb97b1a-5e61-4668-a9c4-31769504ed43","Type":"ContainerStarted","Data":"2120521ac44c1eef68309ad0b1a30b0b6d44a2af6cde5afcdd76f3bc2164b7f1"}
Nov 02 15:30:01.898007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.897988    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" event={"ID":"4a739bb4-d8f5-4b46-a635-d486b4d7e314","Type":"ContainerStarted","Data":"d131efd804b4c484248d96b088b845586223f0e58b82565af19eb9df1fd8d4bc"}
Nov 02 15:30:01.900105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.900073    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" event={"ID":"55911434-6f57-4840-a09a-a67befd78f30","Type":"ContainerStarted","Data":"57a6735b683834a2a7355773df6e3bb992651ef916b7aef317751a8e41dd1df5"}
Nov 02 15:30:01.902200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.902164    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" event={"ID":"70b86377-400c-4ec3-8444-5e5c62e01fae","Type":"ContainerStarted","Data":"e9c79be2595d31dbfca25a40cab1a901e441083579dfe31a99a3e8d4d63581ea"}
Nov 02 15:30:01.902368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.902353    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" event={"ID":"70b86377-400c-4ec3-8444-5e5c62e01fae","Type":"ContainerStarted","Data":"c2de8317256a7c9d45c1ef1af733b58ffeb6e10d645372e009ad68ed87d68d13"}
Nov 02 15:30:01.904693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.904614    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" event={"ID":"921e240b-8fb7-4ea1-baf4-e8e539ade6ef","Type":"ContainerStarted","Data":"e001f9753fb3cf7ca6856a5c754d0e1f381588e95a37d5bde60da3cc9f0a5648"}
Nov 02 15:30:01.908029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.906294    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" event={"ID":"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd","Type":"ContainerStarted","Data":"cabddd835ee45230000f8d6d62266819de65e8f937eeb3cf273b7afc7b4465de"}
Nov 02 15:30:01.908029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.907919    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" event={"ID":"4fd3b658-cf74-49ee-b1d6-75141abc0dd4","Type":"ContainerStarted","Data":"8c75e9e6a2eb56cd76455ace1da95e968accea3f5bff6f6543a657854aea42e0"}
Nov 02 15:30:01.909771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.909744    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" event={"ID":"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85","Type":"ContainerStarted","Data":"79cc2ae5efc32377bda36e18ec218694b334b9d83c40b42f2a5452526571228b"}
Nov 02 15:30:01.911787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.911765    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"]
Nov 02 15:30:01.960327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.960268    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" podStartSLOduration=1.960254103 podStartE2EDuration="1.960254103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:01.957570739 +0000 UTC m=+1996.211524363" watchObservedRunningTime="2026-11-02 15:30:01.960254103 +0000 UTC m=+1996.214207727"
Nov 02 15:30:01.978132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:01.978091    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"]
Nov 02 15:30:02.072868 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:30:02.072816    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cbf4ea_ed62_4534_9619_4e3c2b04d271.slice/crio-5a2f0c8f29c3f8f6a369e31633240e04fe9b1c7f5228c5697f911a3d1d71431d WatchSource:0}: Error finding container 5a2f0c8f29c3f8f6a369e31633240e04fe9b1c7f5228c5697f911a3d1d71431d: Status 404 returned error can't find the container with id 5a2f0c8f29c3f8f6a369e31633240e04fe9b1c7f5228c5697f911a3d1d71431d
Nov 02 15:30:02.857267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.857226    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:02.927744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.927374    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" event={"ID":"921e240b-8fb7-4ea1-baf4-e8e539ade6ef","Type":"ContainerStarted","Data":"fa66e48a2fe7d612ef21d36150da28a110b57967990a1a4af47392273287f0f9"}
Nov 02 15:30:02.929994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.929959    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" event={"ID":"b9cbf4ea-ed62-4534-9619-4e3c2b04d271","Type":"ContainerStarted","Data":"5a2f0c8f29c3f8f6a369e31633240e04fe9b1c7f5228c5697f911a3d1d71431d"}
Nov 02 15:30:02.939193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.939147    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" event={"ID":"a2a57c32-b7d5-440a-adbc-6161142b71db","Type":"ContainerStarted","Data":"7b5da487e22ae0be5f57bbedf4e8604fb6b3c4a29f21a22f045a9e3e0403edac"}
Nov 02 15:30:02.939404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.939223    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" event={"ID":"a2a57c32-b7d5-440a-adbc-6161142b71db","Type":"ContainerStarted","Data":"bf8a9a91a96189fb770cf0308023696435956d3f0b8a1e299c1cf1183c3fb023"}
Nov 02 15:30:02.945099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.945062    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" event={"ID":"4fd3b658-cf74-49ee-b1d6-75141abc0dd4","Type":"ContainerStarted","Data":"e0bd07f6355a65ababb4a7f47376d41ebb7180b5f8705b6d03595242a97e23d8"}
Nov 02 15:30:02.947556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.947523    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" event={"ID":"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85","Type":"ContainerStarted","Data":"f50f817f45e56f7a3062a17da5315fb2d5221ca3983af65e99475180a7bf6566"}
Nov 02 15:30:02.949533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.949502    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" event={"ID":"63ca184a-7c3f-43fb-acb6-dce5d342b804","Type":"ContainerStarted","Data":"c9885007de6f74a003361bc0c17d8c6ba1f8a98cdff69c16b5f984efaef4ddf9"}
Nov 02 15:30:02.951514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.951486    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" event={"ID":"3c84259f-d067-478d-ac0d-4c6087b95a2d","Type":"ContainerStarted","Data":"d9fdcddd787ed383cf18637a44b7d33a2371e1198e935824acf0884927fd47c2"}
Nov 02 15:30:02.953631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.953608    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" event={"ID":"5748e924-6bf8-41b2-b5f6-b9e681456dcb","Type":"ContainerStarted","Data":"e6e01af71e7649eca7c5d8bd4795f93ba075185f200a1d94cd483afd470bad1b"}
Nov 02 15:30:02.955755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.955728    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" event={"ID":"91c69419-8ae3-488b-ae72-137540b1afc8","Type":"ContainerStarted","Data":"d97648efe5d3a7721be251136c1cea7ca839f7a7b2ad00f1304308f05796737d"}
Nov 02 15:30:02.958521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.958497    7019 generic.go:356] "Generic (PLEG): container finished" podID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerID="2f0888f307bf47325488a24caa24bf8382a8ea9c2a316f073c63d671cd03e297" exitCode=60
Nov 02 15:30:02.958638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.958582    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" event={"ID":"1eb97b1a-5e61-4668-a9c4-31769504ed43","Type":"ContainerDied","Data":"2f0888f307bf47325488a24caa24bf8382a8ea9c2a316f073c63d671cd03e297"}
Nov 02 15:30:02.959076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.959053    7019 scope.go:122] "RemoveContainer" containerID="2f0888f307bf47325488a24caa24bf8382a8ea9c2a316f073c63d671cd03e297"
Nov 02 15:30:02.964072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.964021    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" event={"ID":"55911434-6f57-4840-a09a-a67befd78f30","Type":"ContainerStarted","Data":"be0bead4d4c6774bb09af4364a7a299eec7950c0f7d5feca14d3ef458e91a347"}
Nov 02 15:30:02.980487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.980438    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" podStartSLOduration=2.980425485 podStartE2EDuration="2.980425485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.975795912 +0000 UTC m=+1997.229749537" watchObservedRunningTime="2026-11-02 15:30:02.980425485 +0000 UTC m=+1997.234379109"
Nov 02 15:30:02.983920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.983890    7019 generic.go:356] "Generic (PLEG): container finished" podID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerID="e9c79be2595d31dbfca25a40cab1a901e441083579dfe31a99a3e8d4d63581ea" exitCode=60
Nov 02 15:30:02.984382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.984364    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" event={"ID":"70b86377-400c-4ec3-8444-5e5c62e01fae","Type":"ContainerDied","Data":"e9c79be2595d31dbfca25a40cab1a901e441083579dfe31a99a3e8d4d63581ea"}
Nov 02 15:30:02.984582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:02.984568    7019 scope.go:122] "RemoveContainer" containerID="e9c79be2595d31dbfca25a40cab1a901e441083579dfe31a99a3e8d4d63581ea"
Nov 02 15:30:03.021346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.021287    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" podStartSLOduration=3.021270451 podStartE2EDuration="3.021270451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.018855403 +0000 UTC m=+1997.272809029" watchObservedRunningTime="2026-11-02 15:30:03.021270451 +0000 UTC m=+1997.275224077"
Nov 02 15:30:03.072884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.070780    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" podStartSLOduration=3.070760108 podStartE2EDuration="3.070760108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.066094121 +0000 UTC m=+1997.320047746" watchObservedRunningTime="2026-11-02 15:30:03.070760108 +0000 UTC m=+1997.324713733"
Nov 02 15:30:03.121765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.121716    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" podStartSLOduration=3.121701868 podStartE2EDuration="3.121701868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.118497278 +0000 UTC m=+1997.372450901" watchObservedRunningTime="2026-11-02 15:30:03.121701868 +0000 UTC m=+1997.375655489"
Nov 02 15:30:03.164794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.164732    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" podStartSLOduration=3.164715201 podStartE2EDuration="3.164715201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.164584028 +0000 UTC m=+1997.418537652" watchObservedRunningTime="2026-11-02 15:30:03.164715201 +0000 UTC m=+1997.418668828"
Nov 02 15:30:03.211230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.211179    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" podStartSLOduration=3.211166833 podStartE2EDuration="3.211166833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.210303794 +0000 UTC m=+1997.464257418" watchObservedRunningTime="2026-11-02 15:30:03.211166833 +0000 UTC m=+1997.465120458"
Nov 02 15:30:03.305326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.305273    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" podStartSLOduration=3.305260811 podStartE2EDuration="3.305260811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.304013843 +0000 UTC m=+1997.557967467" watchObservedRunningTime="2026-11-02 15:30:03.305260811 +0000 UTC m=+1997.559214434"
Nov 02 15:30:03.353717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.353642    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" podStartSLOduration=3.353628348 podStartE2EDuration="3.353628348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.348772828 +0000 UTC m=+1997.602726452" watchObservedRunningTime="2026-11-02 15:30:03.353628348 +0000 UTC m=+1997.607581965"
Nov 02 15:30:03.390320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.390268    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" podStartSLOduration=3.390253887 podStartE2EDuration="3.390253887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.389785147 +0000 UTC m=+1997.643738781" watchObservedRunningTime="2026-11-02 15:30:03.390253887 +0000 UTC m=+1997.644207511"
Nov 02 15:30:03.433299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:03.433250    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" podStartSLOduration=3.433235367 podStartE2EDuration="3.433235367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:03.430879311 +0000 UTC m=+1997.684832936" watchObservedRunningTime="2026-11-02 15:30:03.433235367 +0000 UTC m=+1997.687188991"
Nov 02 15:30:04.001051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.001014    7019 generic.go:356] "Generic (PLEG): container finished" podID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerID="e6e01af71e7649eca7c5d8bd4795f93ba075185f200a1d94cd483afd470bad1b" exitCode=60
Nov 02 15:30:04.001467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.001112    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" event={"ID":"5748e924-6bf8-41b2-b5f6-b9e681456dcb","Type":"ContainerDied","Data":"e6e01af71e7649eca7c5d8bd4795f93ba075185f200a1d94cd483afd470bad1b"}
Nov 02 15:30:04.001467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.001432    7019 scope.go:122] "RemoveContainer" containerID="e6e01af71e7649eca7c5d8bd4795f93ba075185f200a1d94cd483afd470bad1b"
Nov 02 15:30:04.003300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.003277    7019 generic.go:356] "Generic (PLEG): container finished" podID="91c69419-8ae3-488b-ae72-137540b1afc8" containerID="d97648efe5d3a7721be251136c1cea7ca839f7a7b2ad00f1304308f05796737d" exitCode=60
Nov 02 15:30:04.003424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.003351    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" event={"ID":"91c69419-8ae3-488b-ae72-137540b1afc8","Type":"ContainerDied","Data":"d97648efe5d3a7721be251136c1cea7ca839f7a7b2ad00f1304308f05796737d"}
Nov 02 15:30:04.004056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.003736    7019 scope.go:122] "RemoveContainer" containerID="d97648efe5d3a7721be251136c1cea7ca839f7a7b2ad00f1304308f05796737d"
Nov 02 15:30:04.008831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.008805    7019 generic.go:356] "Generic (PLEG): container finished" podID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerID="13fbe3862907e3c79752a0edc6d5ec8febc402d6085ad002f8254c07c0e87285" exitCode=60
Nov 02 15:30:04.008934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.008867    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" event={"ID":"4a739bb4-d8f5-4b46-a635-d486b4d7e314","Type":"ContainerDied","Data":"13fbe3862907e3c79752a0edc6d5ec8febc402d6085ad002f8254c07c0e87285"}
Nov 02 15:30:04.009271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.009229    7019 scope.go:122] "RemoveContainer" containerID="13fbe3862907e3c79752a0edc6d5ec8febc402d6085ad002f8254c07c0e87285"
Nov 02 15:30:04.010753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.010732    7019 generic.go:356] "Generic (PLEG): container finished" podID="55911434-6f57-4840-a09a-a67befd78f30" containerID="be0bead4d4c6774bb09af4364a7a299eec7950c0f7d5feca14d3ef458e91a347" exitCode=60
Nov 02 15:30:04.010867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.010820    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" event={"ID":"55911434-6f57-4840-a09a-a67befd78f30","Type":"ContainerDied","Data":"be0bead4d4c6774bb09af4364a7a299eec7950c0f7d5feca14d3ef458e91a347"}
Nov 02 15:30:04.011060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.011044    7019 scope.go:122] "RemoveContainer" containerID="be0bead4d4c6774bb09af4364a7a299eec7950c0f7d5feca14d3ef458e91a347"
Nov 02 15:30:04.013095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.013071    7019 generic.go:356] "Generic (PLEG): container finished" podID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerID="fa66e48a2fe7d612ef21d36150da28a110b57967990a1a4af47392273287f0f9" exitCode=60
Nov 02 15:30:04.013198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.013158    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" event={"ID":"921e240b-8fb7-4ea1-baf4-e8e539ade6ef","Type":"ContainerDied","Data":"fa66e48a2fe7d612ef21d36150da28a110b57967990a1a4af47392273287f0f9"}
Nov 02 15:30:04.013388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.013368    7019 scope.go:122] "RemoveContainer" containerID="fa66e48a2fe7d612ef21d36150da28a110b57967990a1a4af47392273287f0f9"
Nov 02 15:30:04.015479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.015309    7019 generic.go:356] "Generic (PLEG): container finished" podID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerID="e7b86edbd48441166587d02e3792345ad44934b83bf7bf2802c06a3327f2a1df" exitCode=60
Nov 02 15:30:04.015479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.015409    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" event={"ID":"b9cbf4ea-ed62-4534-9619-4e3c2b04d271","Type":"ContainerDied","Data":"e7b86edbd48441166587d02e3792345ad44934b83bf7bf2802c06a3327f2a1df"}
Nov 02 15:30:04.015637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.015618    7019 scope.go:122] "RemoveContainer" containerID="e7b86edbd48441166587d02e3792345ad44934b83bf7bf2802c06a3327f2a1df"
Nov 02 15:30:04.018291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.018271    7019 generic.go:356] "Generic (PLEG): container finished" podID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerID="7b5da487e22ae0be5f57bbedf4e8604fb6b3c4a29f21a22f045a9e3e0403edac" exitCode=60
Nov 02 15:30:04.018419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.018395    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" event={"ID":"a2a57c32-b7d5-440a-adbc-6161142b71db","Type":"ContainerDied","Data":"7b5da487e22ae0be5f57bbedf4e8604fb6b3c4a29f21a22f045a9e3e0403edac"}
Nov 02 15:30:04.018559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.018543    7019 scope.go:122] "RemoveContainer" containerID="7b5da487e22ae0be5f57bbedf4e8604fb6b3c4a29f21a22f045a9e3e0403edac"
Nov 02 15:30:04.020204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.020185    7019 generic.go:356] "Generic (PLEG): container finished" podID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerID="11d4be91ed2755b75678cf2c4b16bdf1a0dfcf32cdad59538f671415f1831460" exitCode=60
Nov 02 15:30:04.020299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.020248    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" event={"ID":"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd","Type":"ContainerDied","Data":"11d4be91ed2755b75678cf2c4b16bdf1a0dfcf32cdad59538f671415f1831460"}
Nov 02 15:30:04.020637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.020620    7019 scope.go:122] "RemoveContainer" containerID="11d4be91ed2755b75678cf2c4b16bdf1a0dfcf32cdad59538f671415f1831460"
Nov 02 15:30:04.022600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.022579    7019 generic.go:356] "Generic (PLEG): container finished" podID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerID="e0bd07f6355a65ababb4a7f47376d41ebb7180b5f8705b6d03595242a97e23d8" exitCode=60
Nov 02 15:30:04.023005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.022979    7019 scope.go:122] "RemoveContainer" containerID="e0bd07f6355a65ababb4a7f47376d41ebb7180b5f8705b6d03595242a97e23d8"
Nov 02 15:30:04.023332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.023310    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" event={"ID":"4fd3b658-cf74-49ee-b1d6-75141abc0dd4","Type":"ContainerDied","Data":"e0bd07f6355a65ababb4a7f47376d41ebb7180b5f8705b6d03595242a97e23d8"}
Nov 02 15:30:04.025328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.025306    7019 generic.go:356] "Generic (PLEG): container finished" podID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerID="f50f817f45e56f7a3062a17da5315fb2d5221ca3983af65e99475180a7bf6566" exitCode=60
Nov 02 15:30:04.025414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.025386    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" event={"ID":"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85","Type":"ContainerDied","Data":"f50f817f45e56f7a3062a17da5315fb2d5221ca3983af65e99475180a7bf6566"}
Nov 02 15:30:04.032355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.028950    7019 generic.go:356] "Generic (PLEG): container finished" podID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerID="0165bbb93d43df9787e074aa74d6f490e3a2627f5c34493830a8fba02bcc6dbd" exitCode=60
Nov 02 15:30:04.032355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.029072    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" event={"ID":"63ca184a-7c3f-43fb-acb6-dce5d342b804","Type":"ContainerDied","Data":"0165bbb93d43df9787e074aa74d6f490e3a2627f5c34493830a8fba02bcc6dbd"}
Nov 02 15:30:04.033007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.032902    7019 scope.go:122] "RemoveContainer" containerID="f50f817f45e56f7a3062a17da5315fb2d5221ca3983af65e99475180a7bf6566"
Nov 02 15:30:04.033007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.032988    7019 scope.go:122] "RemoveContainer" containerID="0165bbb93d43df9787e074aa74d6f490e3a2627f5c34493830a8fba02bcc6dbd"
Nov 02 15:30:04.034042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.034019    7019 generic.go:356] "Generic (PLEG): container finished" podID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerID="d9fdcddd787ed383cf18637a44b7d33a2371e1198e935824acf0884927fd47c2" exitCode=60
Nov 02 15:30:04.034150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.034064    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" event={"ID":"3c84259f-d067-478d-ac0d-4c6087b95a2d","Type":"ContainerDied","Data":"d9fdcddd787ed383cf18637a44b7d33a2371e1198e935824acf0884927fd47c2"}
Nov 02 15:30:04.035046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.035020    7019 scope.go:122] "RemoveContainer" containerID="d9fdcddd787ed383cf18637a44b7d33a2371e1198e935824acf0884927fd47c2"
Nov 02 15:30:04.611375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:04.611341    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:05.045832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.045796    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" event={"ID":"921e240b-8fb7-4ea1-baf4-e8e539ade6ef","Type":"ContainerStarted","Data":"c9145d9680880f4230dbe8158ac2fd1e3c0df04ffa03f93643c26ae06f7eae42"}
Nov 02 15:30:05.047948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.047915    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" event={"ID":"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd","Type":"ContainerStarted","Data":"2f0b52e160a8df009704bcd25e4cd09f22755307c470bdc525f8f4bb0f4bbb11"}
Nov 02 15:30:05.050223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.050195    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" event={"ID":"63ca184a-7c3f-43fb-acb6-dce5d342b804","Type":"ContainerStarted","Data":"1de01af613970487622a67b20d33aed21bada2c5d1f275bcd1cf278742b5ea25"}
Nov 02 15:30:05.054237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.054209    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" event={"ID":"5748e924-6bf8-41b2-b5f6-b9e681456dcb","Type":"ContainerStarted","Data":"eb0d69c7a963ffaa8f8e76b44bff2ad8696580ab67621d54de87a078c1b0d75a"}
Nov 02 15:30:05.056118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.056095    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" event={"ID":"91c69419-8ae3-488b-ae72-137540b1afc8","Type":"ContainerStarted","Data":"e259732248977141c706642252e6619addfc417c46af92952249127affd6bab2"}
Nov 02 15:30:05.058177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.058152    7019 generic.go:356] "Generic (PLEG): container finished" podID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerID="c8207278dff95be2c45346165447afe293e218cb56e195e875b1d0afa72f6d26" exitCode=60
Nov 02 15:30:05.058294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.058241    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" event={"ID":"1eb97b1a-5e61-4668-a9c4-31769504ed43","Type":"ContainerDied","Data":"c8207278dff95be2c45346165447afe293e218cb56e195e875b1d0afa72f6d26"}
Nov 02 15:30:05.058353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.058301    7019 scope.go:122] "RemoveContainer" containerID="2f0888f307bf47325488a24caa24bf8382a8ea9c2a316f073c63d671cd03e297"
Nov 02 15:30:05.058480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.058465    7019 scope.go:122] "RemoveContainer" containerID="c8207278dff95be2c45346165447afe293e218cb56e195e875b1d0afa72f6d26"
Nov 02 15:30:05.058830 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:05.058781    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq_odh-ai-gateway-infra(1eb97b1a-5e61-4668-a9c4-31769504ed43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43"
Nov 02 15:30:05.060425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.060405    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" event={"ID":"4a739bb4-d8f5-4b46-a635-d486b4d7e314","Type":"ContainerStarted","Data":"6a0b455d244642c72110805d6b89e547e223de458be1532cd01a1d9c3a52d81e"}
Nov 02 15:30:05.063149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.063124    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" event={"ID":"55911434-6f57-4840-a09a-a67befd78f30","Type":"ContainerStarted","Data":"362892f1d5757928507e7826b8adc2242c319bd04207385b6cb7c07f4f215d63"}
Nov 02 15:30:05.086043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.085759    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" event={"ID":"70b86377-400c-4ec3-8444-5e5c62e01fae","Type":"ContainerStarted","Data":"0ab574e8ef66e00a85c8838de8fb04ca1d6ecc205a258480a507f5d537e5cc72"}
Nov 02 15:30:05.183405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:05.183345    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" podStartSLOduration=5.183325858 podStartE2EDuration="5.183325858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:05.181601915 +0000 UTC m=+1999.435555539" watchObservedRunningTime="2026-11-02 15:30:05.183325858 +0000 UTC m=+1999.437279484"
Nov 02 15:30:06.106181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.106141    7019 generic.go:356] "Generic (PLEG): container finished" podID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerID="f05a9eb32a0ab320a5a0c688f70f6b007c764ad9d764134dd0dec0e3f0ef2c70" exitCode=60
Nov 02 15:30:06.106792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.106240    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" event={"ID":"3c84259f-d067-478d-ac0d-4c6087b95a2d","Type":"ContainerDied","Data":"f05a9eb32a0ab320a5a0c688f70f6b007c764ad9d764134dd0dec0e3f0ef2c70"}
Nov 02 15:30:06.106792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.106334    7019 scope.go:122] "RemoveContainer" containerID="d9fdcddd787ed383cf18637a44b7d33a2371e1198e935824acf0884927fd47c2"
Nov 02 15:30:06.106792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.106769    7019 scope.go:122] "RemoveContainer" containerID="f05a9eb32a0ab320a5a0c688f70f6b007c764ad9d764134dd0dec0e3f0ef2c70"
Nov 02 15:30:06.107182 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.107156    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f_odh-ai-gateway-infra(3c84259f-d067-478d-ac0d-4c6087b95a2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d"
Nov 02 15:30:06.108608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.108583    7019 generic.go:356] "Generic (PLEG): container finished" podID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerID="eb0d69c7a963ffaa8f8e76b44bff2ad8696580ab67621d54de87a078c1b0d75a" exitCode=60
Nov 02 15:30:06.108719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.108682    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" event={"ID":"5748e924-6bf8-41b2-b5f6-b9e681456dcb","Type":"ContainerDied","Data":"eb0d69c7a963ffaa8f8e76b44bff2ad8696580ab67621d54de87a078c1b0d75a"}
Nov 02 15:30:06.108989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.108973    7019 scope.go:122] "RemoveContainer" containerID="eb0d69c7a963ffaa8f8e76b44bff2ad8696580ab67621d54de87a078c1b0d75a"
Nov 02 15:30:06.109337 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.109312    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c_odh-ai-gateway-infra(5748e924-6bf8-41b2-b5f6-b9e681456dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb"
Nov 02 15:30:06.110840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.110820    7019 generic.go:356] "Generic (PLEG): container finished" podID="91c69419-8ae3-488b-ae72-137540b1afc8" containerID="e259732248977141c706642252e6619addfc417c46af92952249127affd6bab2" exitCode=60
Nov 02 15:30:06.110965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.110883    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" event={"ID":"91c69419-8ae3-488b-ae72-137540b1afc8","Type":"ContainerDied","Data":"e259732248977141c706642252e6619addfc417c46af92952249127affd6bab2"}
Nov 02 15:30:06.111149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.111133    7019 scope.go:122] "RemoveContainer" containerID="e259732248977141c706642252e6619addfc417c46af92952249127affd6bab2"
Nov 02 15:30:06.111369 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.111349    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855_odh-ai-gateway-infra(91c69419-8ae3-488b-ae72-137540b1afc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" podUID="91c69419-8ae3-488b-ae72-137540b1afc8"
Nov 02 15:30:06.112899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.112882    7019 scope.go:122] "RemoveContainer" containerID="c8207278dff95be2c45346165447afe293e218cb56e195e875b1d0afa72f6d26"
Nov 02 15:30:06.113142 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.113119    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq_odh-ai-gateway-infra(1eb97b1a-5e61-4668-a9c4-31769504ed43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43"
Nov 02 15:30:06.114720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.114701    7019 generic.go:356] "Generic (PLEG): container finished" podID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerID="6a0b455d244642c72110805d6b89e547e223de458be1532cd01a1d9c3a52d81e" exitCode=60
Nov 02 15:30:06.114805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.114770    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" event={"ID":"4a739bb4-d8f5-4b46-a635-d486b4d7e314","Type":"ContainerDied","Data":"6a0b455d244642c72110805d6b89e547e223de458be1532cd01a1d9c3a52d81e"}
Nov 02 15:30:06.115056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.115041    7019 scope.go:122] "RemoveContainer" containerID="6a0b455d244642c72110805d6b89e547e223de458be1532cd01a1d9c3a52d81e"
Nov 02 15:30:06.115518 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.115470    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn_odh-ai-gateway-infra(4a739bb4-d8f5-4b46-a635-d486b4d7e314)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314"
Nov 02 15:30:06.116628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.116609    7019 generic.go:356] "Generic (PLEG): container finished" podID="55911434-6f57-4840-a09a-a67befd78f30" containerID="362892f1d5757928507e7826b8adc2242c319bd04207385b6cb7c07f4f215d63" exitCode=60
Nov 02 15:30:06.116721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.116689    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" event={"ID":"55911434-6f57-4840-a09a-a67befd78f30","Type":"ContainerDied","Data":"362892f1d5757928507e7826b8adc2242c319bd04207385b6cb7c07f4f215d63"}
Nov 02 15:30:06.116877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.116859    7019 scope.go:122] "RemoveContainer" containerID="362892f1d5757928507e7826b8adc2242c319bd04207385b6cb7c07f4f215d63"
Nov 02 15:30:06.117091 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.117072    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8_odh-ai-gateway-infra(55911434-6f57-4840-a09a-a67befd78f30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" podUID="55911434-6f57-4840-a09a-a67befd78f30"
Nov 02 15:30:06.118611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.118587    7019 generic.go:356] "Generic (PLEG): container finished" podID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerID="0ab574e8ef66e00a85c8838de8fb04ca1d6ecc205a258480a507f5d537e5cc72" exitCode=60
Nov 02 15:30:06.118732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.118680    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" event={"ID":"70b86377-400c-4ec3-8444-5e5c62e01fae","Type":"ContainerDied","Data":"0ab574e8ef66e00a85c8838de8fb04ca1d6ecc205a258480a507f5d537e5cc72"}
Nov 02 15:30:06.118945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.118926    7019 scope.go:122] "RemoveContainer" containerID="0ab574e8ef66e00a85c8838de8fb04ca1d6ecc205a258480a507f5d537e5cc72"
Nov 02 15:30:06.119201 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.119180    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc_odh-ai-gateway-infra(70b86377-400c-4ec3-8444-5e5c62e01fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae"
Nov 02 15:30:06.120911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.120888    7019 generic.go:356] "Generic (PLEG): container finished" podID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerID="c9145d9680880f4230dbe8158ac2fd1e3c0df04ffa03f93643c26ae06f7eae42" exitCode=60
Nov 02 15:30:06.120986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.120956    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" event={"ID":"921e240b-8fb7-4ea1-baf4-e8e539ade6ef","Type":"ContainerDied","Data":"c9145d9680880f4230dbe8158ac2fd1e3c0df04ffa03f93643c26ae06f7eae42"}
Nov 02 15:30:06.121178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.121162    7019 scope.go:122] "RemoveContainer" containerID="c9145d9680880f4230dbe8158ac2fd1e3c0df04ffa03f93643c26ae06f7eae42"
Nov 02 15:30:06.121432 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.121406    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b_odh-ai-gateway-infra(921e240b-8fb7-4ea1-baf4-e8e539ade6ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef"
Nov 02 15:30:06.122980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.122957    7019 generic.go:356] "Generic (PLEG): container finished" podID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerID="5dbdde4a80062c07df25c843097e573cd9d56b7a67238d0d3b7efa4ace1ec635" exitCode=60
Nov 02 15:30:06.123071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.122982    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" event={"ID":"b9cbf4ea-ed62-4534-9619-4e3c2b04d271","Type":"ContainerDied","Data":"5dbdde4a80062c07df25c843097e573cd9d56b7a67238d0d3b7efa4ace1ec635"}
Nov 02 15:30:06.123277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.123259    7019 scope.go:122] "RemoveContainer" containerID="5dbdde4a80062c07df25c843097e573cd9d56b7a67238d0d3b7efa4ace1ec635"
Nov 02 15:30:06.123497 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.123477    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp_odh-ai-gateway-infra(b9cbf4ea-ed62-4534-9619-4e3c2b04d271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271"
Nov 02 15:30:06.125240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.125218    7019 generic.go:356] "Generic (PLEG): container finished" podID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerID="31e4bc6cec647a80d38073962c39f51dece1fbe43dfb48fe543ae3ee929bbcf0" exitCode=60
Nov 02 15:30:06.125336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.125295    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" event={"ID":"a2a57c32-b7d5-440a-adbc-6161142b71db","Type":"ContainerDied","Data":"31e4bc6cec647a80d38073962c39f51dece1fbe43dfb48fe543ae3ee929bbcf0"}
Nov 02 15:30:06.125555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.125535    7019 scope.go:122] "RemoveContainer" containerID="31e4bc6cec647a80d38073962c39f51dece1fbe43dfb48fe543ae3ee929bbcf0"
Nov 02 15:30:06.125775 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.125756    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4_odh-ai-gateway-infra(a2a57c32-b7d5-440a-adbc-6161142b71db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db"
Nov 02 15:30:06.127421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.127403    7019 generic.go:356] "Generic (PLEG): container finished" podID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerID="2f0b52e160a8df009704bcd25e4cd09f22755307c470bdc525f8f4bb0f4bbb11" exitCode=60
Nov 02 15:30:06.127507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.127485    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" event={"ID":"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd","Type":"ContainerDied","Data":"2f0b52e160a8df009704bcd25e4cd09f22755307c470bdc525f8f4bb0f4bbb11"}
Nov 02 15:30:06.127907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.127887    7019 scope.go:122] "RemoveContainer" containerID="2f0b52e160a8df009704bcd25e4cd09f22755307c470bdc525f8f4bb0f4bbb11"
Nov 02 15:30:06.128133 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.128115    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg_odh-ai-gateway-infra(efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd"
Nov 02 15:30:06.128278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.128259    7019 scope.go:122] "RemoveContainer" containerID="e6e01af71e7649eca7c5d8bd4795f93ba075185f200a1d94cd483afd470bad1b"
Nov 02 15:30:06.129669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.129633    7019 generic.go:356] "Generic (PLEG): container finished" podID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerID="891ab77424012fc3eedd407e8e8a191b2ab6aaec1e993637a75827fa8fda85b6" exitCode=60
Nov 02 15:30:06.129761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.129695    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" event={"ID":"4fd3b658-cf74-49ee-b1d6-75141abc0dd4","Type":"ContainerDied","Data":"891ab77424012fc3eedd407e8e8a191b2ab6aaec1e993637a75827fa8fda85b6"}
Nov 02 15:30:06.129958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.129942    7019 scope.go:122] "RemoveContainer" containerID="891ab77424012fc3eedd407e8e8a191b2ab6aaec1e993637a75827fa8fda85b6"
Nov 02 15:30:06.130204 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.130179    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8_odh-ai-gateway-infra(4fd3b658-cf74-49ee-b1d6-75141abc0dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4"
Nov 02 15:30:06.131596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.131577    7019 generic.go:356] "Generic (PLEG): container finished" podID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerID="59f96dade6af768980077b7b1656fb311b0305b88d6befddce77fd87a710722f" exitCode=60
Nov 02 15:30:06.131727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.131671    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" event={"ID":"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85","Type":"ContainerDied","Data":"59f96dade6af768980077b7b1656fb311b0305b88d6befddce77fd87a710722f"}
Nov 02 15:30:06.131957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.131941    7019 scope.go:122] "RemoveContainer" containerID="59f96dade6af768980077b7b1656fb311b0305b88d6befddce77fd87a710722f"
Nov 02 15:30:06.132202 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.132180    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f_odh-ai-gateway-infra(c8eb9b49-3ae4-4687-a406-a4c35d1dfd85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85"
Nov 02 15:30:06.133719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.133697    7019 generic.go:356] "Generic (PLEG): container finished" podID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerID="1de01af613970487622a67b20d33aed21bada2c5d1f275bcd1cf278742b5ea25" exitCode=60
Nov 02 15:30:06.133808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.133743    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" event={"ID":"63ca184a-7c3f-43fb-acb6-dce5d342b804","Type":"ContainerDied","Data":"1de01af613970487622a67b20d33aed21bada2c5d1f275bcd1cf278742b5ea25"}
Nov 02 15:30:06.133937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.133926    7019 scope.go:122] "RemoveContainer" containerID="1de01af613970487622a67b20d33aed21bada2c5d1f275bcd1cf278742b5ea25"
Nov 02 15:30:06.134130 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:06.134111    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd_odh-ai-gateway-infra(63ca184a-7c3f-43fb-acb6-dce5d342b804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804"
Nov 02 15:30:06.141867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.141852    7019 scope.go:122] "RemoveContainer" containerID="d97648efe5d3a7721be251136c1cea7ca839f7a7b2ad00f1304308f05796737d"
Nov 02 15:30:06.158745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.158721    7019 scope.go:122] "RemoveContainer" containerID="13fbe3862907e3c79752a0edc6d5ec8febc402d6085ad002f8254c07c0e87285"
Nov 02 15:30:06.173770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.173742    7019 scope.go:122] "RemoveContainer" containerID="be0bead4d4c6774bb09af4364a7a299eec7950c0f7d5feca14d3ef458e91a347"
Nov 02 15:30:06.186392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.186370    7019 scope.go:122] "RemoveContainer" containerID="e9c79be2595d31dbfca25a40cab1a901e441083579dfe31a99a3e8d4d63581ea"
Nov 02 15:30:06.200463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.200441    7019 scope.go:122] "RemoveContainer" containerID="fa66e48a2fe7d612ef21d36150da28a110b57967990a1a4af47392273287f0f9"
Nov 02 15:30:06.213951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.213930    7019 scope.go:122] "RemoveContainer" containerID="e7b86edbd48441166587d02e3792345ad44934b83bf7bf2802c06a3327f2a1df"
Nov 02 15:30:06.229212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.229182    7019 scope.go:122] "RemoveContainer" containerID="7b5da487e22ae0be5f57bbedf4e8604fb6b3c4a29f21a22f045a9e3e0403edac"
Nov 02 15:30:06.242424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.242398    7019 scope.go:122] "RemoveContainer" containerID="11d4be91ed2755b75678cf2c4b16bdf1a0dfcf32cdad59538f671415f1831460"
Nov 02 15:30:06.256351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.256326    7019 scope.go:122] "RemoveContainer" containerID="e0bd07f6355a65ababb4a7f47376d41ebb7180b5f8705b6d03595242a97e23d8"
Nov 02 15:30:06.269884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.269860    7019 scope.go:122] "RemoveContainer" containerID="f50f817f45e56f7a3062a17da5315fb2d5221ca3983af65e99475180a7bf6566"
Nov 02 15:30:06.282238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.282218    7019 scope.go:122] "RemoveContainer" containerID="0165bbb93d43df9787e074aa74d6f490e3a2627f5c34493830a8fba02bcc6dbd"
Nov 02 15:30:06.405399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:06.405367    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:07.145185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.145156    7019 scope.go:122] "RemoveContainer" containerID="f05a9eb32a0ab320a5a0c688f70f6b007c764ad9d764134dd0dec0e3f0ef2c70"
Nov 02 15:30:07.146080 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.146050    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f_odh-ai-gateway-infra(3c84259f-d067-478d-ac0d-4c6087b95a2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d"
Nov 02 15:30:07.147051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.147031    7019 scope.go:122] "RemoveContainer" containerID="eb0d69c7a963ffaa8f8e76b44bff2ad8696580ab67621d54de87a078c1b0d75a"
Nov 02 15:30:07.147345 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.147320    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c_odh-ai-gateway-infra(5748e924-6bf8-41b2-b5f6-b9e681456dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb"
Nov 02 15:30:07.148768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.148750    7019 scope.go:122] "RemoveContainer" containerID="e259732248977141c706642252e6619addfc417c46af92952249127affd6bab2"
Nov 02 15:30:07.149012 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.148988    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855_odh-ai-gateway-infra(91c69419-8ae3-488b-ae72-137540b1afc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" podUID="91c69419-8ae3-488b-ae72-137540b1afc8"
Nov 02 15:30:07.152101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.152084    7019 scope.go:122] "RemoveContainer" containerID="362892f1d5757928507e7826b8adc2242c319bd04207385b6cb7c07f4f215d63"
Nov 02 15:30:07.152366 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.152339    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8_odh-ai-gateway-infra(55911434-6f57-4840-a09a-a67befd78f30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" podUID="55911434-6f57-4840-a09a-a67befd78f30"
Nov 02 15:30:07.153703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.153686    7019 scope.go:122] "RemoveContainer" containerID="0ab574e8ef66e00a85c8838de8fb04ca1d6ecc205a258480a507f5d537e5cc72"
Nov 02 15:30:07.153911 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.153892    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc_odh-ai-gateway-infra(70b86377-400c-4ec3-8444-5e5c62e01fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae"
Nov 02 15:30:07.155318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.155299    7019 scope.go:122] "RemoveContainer" containerID="c9145d9680880f4230dbe8158ac2fd1e3c0df04ffa03f93643c26ae06f7eae42"
Nov 02 15:30:07.155559 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.155537    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b_odh-ai-gateway-infra(921e240b-8fb7-4ea1-baf4-e8e539ade6ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef"
Nov 02 15:30:07.156940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.156921    7019 scope.go:122] "RemoveContainer" containerID="5dbdde4a80062c07df25c843097e573cd9d56b7a67238d0d3b7efa4ace1ec635"
Nov 02 15:30:07.157147 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.157126    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp_odh-ai-gateway-infra(b9cbf4ea-ed62-4534-9619-4e3c2b04d271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271"
Nov 02 15:30:07.158796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.158777    7019 scope.go:122] "RemoveContainer" containerID="31e4bc6cec647a80d38073962c39f51dece1fbe43dfb48fe543ae3ee929bbcf0"
Nov 02 15:30:07.159051 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.159026    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4_odh-ai-gateway-infra(a2a57c32-b7d5-440a-adbc-6161142b71db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db"
Nov 02 15:30:07.163013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.162994    7019 scope.go:122] "RemoveContainer" containerID="891ab77424012fc3eedd407e8e8a191b2ab6aaec1e993637a75827fa8fda85b6"
Nov 02 15:30:07.163201 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.163184    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8_odh-ai-gateway-infra(4fd3b658-cf74-49ee-b1d6-75141abc0dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4"
Nov 02 15:30:07.164614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.164599    7019 scope.go:122] "RemoveContainer" containerID="59f96dade6af768980077b7b1656fb311b0305b88d6befddce77fd87a710722f"
Nov 02 15:30:07.165012 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.164824    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f_odh-ai-gateway-infra(c8eb9b49-3ae4-4687-a406-a4c35d1dfd85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85"
Nov 02 15:30:07.166269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:07.166252    7019 scope.go:122] "RemoveContainer" containerID="1de01af613970487622a67b20d33aed21bada2c5d1f275bcd1cf278742b5ea25"
Nov 02 15:30:07.166463 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:07.166446    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd_odh-ai-gateway-infra(63ca184a-7c3f-43fb-acb6-dce5d342b804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804"
Nov 02 15:30:08.075217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:08.075174    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:09.527742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:09.527709    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:30:09.540617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:09.540586    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:30:09.637614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:09.637578    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:10.208019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:10.207989    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:30:11.156189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:11.156133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:12.846149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:12.846109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:14.438134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:14.438097    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:15.862275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:15.862246    7019 scope.go:122] "RemoveContainer" containerID="5dbdde4a80062c07df25c843097e573cd9d56b7a67238d0d3b7efa4ace1ec635"
Nov 02 15:30:15.862741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:15.862416    7019 scope.go:122] "RemoveContainer" containerID="eb0d69c7a963ffaa8f8e76b44bff2ad8696580ab67621d54de87a078c1b0d75a"
Nov 02 15:30:15.862741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:15.862576    7019 scope.go:122] "RemoveContainer" containerID="31e4bc6cec647a80d38073962c39f51dece1fbe43dfb48fe543ae3ee929bbcf0"
Nov 02 15:30:16.096686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:16.096640    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:16.745094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:16.744996    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:16.855323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:16.855295    7019 scope.go:122] "RemoveContainer" containerID="891ab77424012fc3eedd407e8e8a191b2ab6aaec1e993637a75827fa8fda85b6"
Nov 02 15:30:16.855500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:16.855422    7019 scope.go:122] "RemoveContainer" containerID="2f0b52e160a8df009704bcd25e4cd09f22755307c470bdc525f8f4bb0f4bbb11"
Nov 02 15:30:16.911052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:16.911018    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:17.272704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.272669    7019 generic.go:356] "Generic (PLEG): container finished" podID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerID="27a6acced39cf50608936f2f039c99332f1a3738c2d8fe2c940e5712686ab857" exitCode=60
Nov 02 15:30:17.272704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.272702    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" event={"ID":"a2a57c32-b7d5-440a-adbc-6161142b71db","Type":"ContainerDied","Data":"27a6acced39cf50608936f2f039c99332f1a3738c2d8fe2c940e5712686ab857"}
Nov 02 15:30:17.272944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.272763    7019 scope.go:122] "RemoveContainer" containerID="31e4bc6cec647a80d38073962c39f51dece1fbe43dfb48fe543ae3ee929bbcf0"
Nov 02 15:30:17.273255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.273229    7019 scope.go:122] "RemoveContainer" containerID="27a6acced39cf50608936f2f039c99332f1a3738c2d8fe2c940e5712686ab857"
Nov 02 15:30:17.273812 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:17.273753    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4_odh-ai-gateway-infra(a2a57c32-b7d5-440a-adbc-6161142b71db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db"
Nov 02 15:30:17.275908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.275886    7019 generic.go:356] "Generic (PLEG): container finished" podID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerID="e567ad8e6966115a449183873d83225ee0ecb2505abb3954d850813b05f41425" exitCode=60
Nov 02 15:30:17.276008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.275971    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" event={"ID":"5748e924-6bf8-41b2-b5f6-b9e681456dcb","Type":"ContainerDied","Data":"e567ad8e6966115a449183873d83225ee0ecb2505abb3954d850813b05f41425"}
Nov 02 15:30:17.276486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.276465    7019 scope.go:122] "RemoveContainer" containerID="e567ad8e6966115a449183873d83225ee0ecb2505abb3954d850813b05f41425"
Nov 02 15:30:17.277325 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:17.277304    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c_odh-ai-gateway-infra(5748e924-6bf8-41b2-b5f6-b9e681456dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb"
Nov 02 15:30:17.278336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.278309    7019 generic.go:356] "Generic (PLEG): container finished" podID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerID="13f081b7dd60a507c7019006fd58839b7647740a424c493bbfc231d781431cf9" exitCode=60
Nov 02 15:30:17.278435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.278354    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" event={"ID":"b9cbf4ea-ed62-4534-9619-4e3c2b04d271","Type":"ContainerDied","Data":"13f081b7dd60a507c7019006fd58839b7647740a424c493bbfc231d781431cf9"}
Nov 02 15:30:17.278795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.278776    7019 scope.go:122] "RemoveContainer" containerID="13f081b7dd60a507c7019006fd58839b7647740a424c493bbfc231d781431cf9"
Nov 02 15:30:17.279145 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:17.279125    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp_odh-ai-gateway-infra(b9cbf4ea-ed62-4534-9619-4e3c2b04d271)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271"
Nov 02 15:30:17.293239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.293208    7019 scope.go:122] "RemoveContainer" containerID="eb0d69c7a963ffaa8f8e76b44bff2ad8696580ab67621d54de87a078c1b0d75a"
Nov 02 15:30:17.310119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.310099    7019 scope.go:122] "RemoveContainer" containerID="5dbdde4a80062c07df25c843097e573cd9d56b7a67238d0d3b7efa4ace1ec635"
Nov 02 15:30:17.844184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.838209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:17.883857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.883824    7019 scope.go:122] "RemoveContainer" containerID="f05a9eb32a0ab320a5a0c688f70f6b007c764ad9d764134dd0dec0e3f0ef2c70"
Nov 02 15:30:17.886567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.885449    7019 scope.go:122] "RemoveContainer" containerID="1de01af613970487622a67b20d33aed21bada2c5d1f275bcd1cf278742b5ea25"
Nov 02 15:30:17.886567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:17.886367    7019 scope.go:122] "RemoveContainer" containerID="e259732248977141c706642252e6619addfc417c46af92952249127affd6bab2"
Nov 02 15:30:18.105741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.105589    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:18.296165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.296129    7019 generic.go:356] "Generic (PLEG): container finished" podID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerID="2bd3f5b242b8b5045e6f07bd7a029712eee9a2c396c36e488ffe3873640b3621" exitCode=60
Nov 02 15:30:18.296310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.296161    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" event={"ID":"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd","Type":"ContainerDied","Data":"2bd3f5b242b8b5045e6f07bd7a029712eee9a2c396c36e488ffe3873640b3621"}
Nov 02 15:30:18.296310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.296219    7019 scope.go:122] "RemoveContainer" containerID="2f0b52e160a8df009704bcd25e4cd09f22755307c470bdc525f8f4bb0f4bbb11"
Nov 02 15:30:18.296777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.296738    7019 scope.go:122] "RemoveContainer" containerID="2bd3f5b242b8b5045e6f07bd7a029712eee9a2c396c36e488ffe3873640b3621"
Nov 02 15:30:18.299039 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:18.297189    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg_odh-ai-gateway-infra(efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd"
Nov 02 15:30:18.299039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.298714    7019 generic.go:356] "Generic (PLEG): container finished" podID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerID="dbfb48a9e76a12ead2527447d3ee01d9a09d06fc957569b7da9ca1545ac21c20" exitCode=60
Nov 02 15:30:18.299039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.298782    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" event={"ID":"4fd3b658-cf74-49ee-b1d6-75141abc0dd4","Type":"ContainerDied","Data":"dbfb48a9e76a12ead2527447d3ee01d9a09d06fc957569b7da9ca1545ac21c20"}
Nov 02 15:30:18.299437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.299226    7019 scope.go:122] "RemoveContainer" containerID="dbfb48a9e76a12ead2527447d3ee01d9a09d06fc957569b7da9ca1545ac21c20"
Nov 02 15:30:18.299824 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:18.299798    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8_odh-ai-gateway-infra(4fd3b658-cf74-49ee-b1d6-75141abc0dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4"
Nov 02 15:30:18.313201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.313180    7019 scope.go:122] "RemoveContainer" containerID="891ab77424012fc3eedd407e8e8a191b2ab6aaec1e993637a75827fa8fda85b6"
Nov 02 15:30:18.343948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.343919    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"]
Nov 02 15:30:18.409018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.408984    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"]
Nov 02 15:30:18.493450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.492873    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"]
Nov 02 15:30:18.634783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.629888    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:18.652887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.652825    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:18.656716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.656693    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a57c32-b7d5-440a-adbc-6161142b71db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a57c32-b7d5-440a-adbc-6161142b71db-maas-api-service-ca\") pod \"a2a57c32-b7d5-440a-adbc-6161142b71db\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") "
Nov 02 15:30:18.657044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.657023    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a57c32-b7d5-440a-adbc-6161142b71db-maas-api-service-ca" pod "a2a57c32-b7d5-440a-adbc-6161142b71db" (UID: "a2a57c32-b7d5-440a-adbc-6161142b71db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:18.657146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.657131    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a57c32-b7d5-440a-adbc-6161142b71db-kube-api-access-jhzbc\" (UniqueName: \"kubernetes.io/projected/a2a57c32-b7d5-440a-adbc-6161142b71db-kube-api-access-jhzbc\") pod \"a2a57c32-b7d5-440a-adbc-6161142b71db\" (UID: \"a2a57c32-b7d5-440a-adbc-6161142b71db\") "
Nov 02 15:30:18.657838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.657822    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a57c32-b7d5-440a-adbc-6161142b71db-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:18.659145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.659126    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a57c32-b7d5-440a-adbc-6161142b71db-kube-api-access-jhzbc" pod "a2a57c32-b7d5-440a-adbc-6161142b71db" (UID: "a2a57c32-b7d5-440a-adbc-6161142b71db"). InnerVolumeSpecName "kube-api-access-jhzbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:18.759056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.759017    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5748e924-6bf8-41b2-b5f6-b9e681456dcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5748e924-6bf8-41b2-b5f6-b9e681456dcb-maas-api-service-ca\") pod \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") "
Nov 02 15:30:18.759236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.759126    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5748e924-6bf8-41b2-b5f6-b9e681456dcb-kube-api-access-qmw2n\" (UniqueName: \"kubernetes.io/projected/5748e924-6bf8-41b2-b5f6-b9e681456dcb-kube-api-access-qmw2n\") pod \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\" (UID: \"5748e924-6bf8-41b2-b5f6-b9e681456dcb\") "
Nov 02 15:30:18.759443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.759417    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5748e924-6bf8-41b2-b5f6-b9e681456dcb-maas-api-service-ca" pod "5748e924-6bf8-41b2-b5f6-b9e681456dcb" (UID: "5748e924-6bf8-41b2-b5f6-b9e681456dcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:18.760003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.759980    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhzbc\" (UniqueName: \"kubernetes.io/projected/a2a57c32-b7d5-440a-adbc-6161142b71db-kube-api-access-jhzbc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:18.760118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.760008    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5748e924-6bf8-41b2-b5f6-b9e681456dcb-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:18.762301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.762267    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5748e924-6bf8-41b2-b5f6-b9e681456dcb-kube-api-access-qmw2n" pod "5748e924-6bf8-41b2-b5f6-b9e681456dcb" (UID: "5748e924-6bf8-41b2-b5f6-b9e681456dcb"). InnerVolumeSpecName "kube-api-access-qmw2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:18.816162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.816130    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:18.855601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.855559    7019 scope.go:122] "RemoveContainer" containerID="362892f1d5757928507e7826b8adc2242c319bd04207385b6cb7c07f4f215d63"
Nov 02 15:30:18.855807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.855784    7019 scope.go:122] "RemoveContainer" containerID="c8207278dff95be2c45346165447afe293e218cb56e195e875b1d0afa72f6d26"
Nov 02 15:30:18.855951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.855933    7019 scope.go:122] "RemoveContainer" containerID="0ab574e8ef66e00a85c8838de8fb04ca1d6ecc205a258480a507f5d537e5cc72"
Nov 02 15:30:18.856097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.856077    7019 scope.go:122] "RemoveContainer" containerID="59f96dade6af768980077b7b1656fb311b0305b88d6befddce77fd87a710722f"
Nov 02 15:30:18.861211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.861186    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-maas-api-service-ca\") pod \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") "
Nov 02 15:30:18.861421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.861402    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-kube-api-access-9bp4p\" (UniqueName: \"kubernetes.io/projected/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-kube-api-access-9bp4p\") pod \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\" (UID: \"b9cbf4ea-ed62-4534-9619-4e3c2b04d271\") "
Nov 02 15:30:18.861996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.861968    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmw2n\" (UniqueName: \"kubernetes.io/projected/5748e924-6bf8-41b2-b5f6-b9e681456dcb-kube-api-access-qmw2n\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:18.862222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.862193    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-maas-api-service-ca" pod "b9cbf4ea-ed62-4534-9619-4e3c2b04d271" (UID: "b9cbf4ea-ed62-4534-9619-4e3c2b04d271"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:18.864109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.864082    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-kube-api-access-9bp4p" pod "b9cbf4ea-ed62-4534-9619-4e3c2b04d271" (UID: "b9cbf4ea-ed62-4534-9619-4e3c2b04d271"). InnerVolumeSpecName "kube-api-access-9bp4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:18.977242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.977193    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bp4p\" (UniqueName: \"kubernetes.io/projected/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-kube-api-access-9bp4p\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:18.977242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:18.977228    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9cbf4ea-ed62-4534-9619-4e3c2b04d271-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.315574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.315504    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4" event={"ID":"a2a57c32-b7d5-440a-adbc-6161142b71db","Type":"ContainerDied","Data":"bf8a9a91a96189fb770cf0308023696435956d3f0b8a1e299c1cf1183c3fb023"}
Nov 02 15:30:19.316243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.316209    7019 scope.go:122] "RemoveContainer" containerID="27a6acced39cf50608936f2f039c99332f1a3738c2d8fe2c940e5712686ab857"
Nov 02 15:30:19.316588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.316514    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"
Nov 02 15:30:19.329614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.329591    7019 generic.go:356] "Generic (PLEG): container finished" podID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerID="1b0287f8cb263e4f0376dae934c89a71611a483941fd32d87c19adccbf983f53" exitCode=60
Nov 02 15:30:19.329735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.329671    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" event={"ID":"63ca184a-7c3f-43fb-acb6-dce5d342b804","Type":"ContainerDied","Data":"1b0287f8cb263e4f0376dae934c89a71611a483941fd32d87c19adccbf983f53"}
Nov 02 15:30:19.330282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.330263    7019 scope.go:122] "RemoveContainer" containerID="1b0287f8cb263e4f0376dae934c89a71611a483941fd32d87c19adccbf983f53"
Nov 02 15:30:19.330709 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:19.330685    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd_odh-ai-gateway-infra(63ca184a-7c3f-43fb-acb6-dce5d342b804)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804"
Nov 02 15:30:19.331559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.331540    7019 generic.go:356] "Generic (PLEG): container finished" podID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerID="2cc056a75b1aa275c90a77681fd7c9aeaf5a5f74857c67d2bc015f084303c054" exitCode=60
Nov 02 15:30:19.331698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.331607    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" event={"ID":"3c84259f-d067-478d-ac0d-4c6087b95a2d","Type":"ContainerDied","Data":"2cc056a75b1aa275c90a77681fd7c9aeaf5a5f74857c67d2bc015f084303c054"}
Nov 02 15:30:19.332002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.331986    7019 scope.go:122] "RemoveContainer" containerID="2cc056a75b1aa275c90a77681fd7c9aeaf5a5f74857c67d2bc015f084303c054"
Nov 02 15:30:19.332370 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:19.332347    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f_odh-ai-gateway-infra(3c84259f-d067-478d-ac0d-4c6087b95a2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d"
Nov 02 15:30:19.333074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.333059    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"
Nov 02 15:30:19.333074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.333067    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c" event={"ID":"5748e924-6bf8-41b2-b5f6-b9e681456dcb","Type":"ContainerDied","Data":"7a367ed94ebc96d9cf3b4838f3ade0fc15715c688daf993c6747225c7d49a5a8"}
Nov 02 15:30:19.334925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.334906    7019 generic.go:356] "Generic (PLEG): container finished" podID="91c69419-8ae3-488b-ae72-137540b1afc8" containerID="916f95c3139d4465bd2a0dcc7b988649c323557d61639e78811f5a15867757ed" exitCode=60
Nov 02 15:30:19.334994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.334977    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" event={"ID":"91c69419-8ae3-488b-ae72-137540b1afc8","Type":"ContainerDied","Data":"916f95c3139d4465bd2a0dcc7b988649c323557d61639e78811f5a15867757ed"}
Nov 02 15:30:19.335566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.335545    7019 scope.go:122] "RemoveContainer" containerID="916f95c3139d4465bd2a0dcc7b988649c323557d61639e78811f5a15867757ed"
Nov 02 15:30:19.335898 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:19.335871    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855_odh-ai-gateway-infra(91c69419-8ae3-488b-ae72-137540b1afc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" podUID="91c69419-8ae3-488b-ae72-137540b1afc8"
Nov 02 15:30:19.336480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.336462    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp" event={"ID":"b9cbf4ea-ed62-4534-9619-4e3c2b04d271","Type":"ContainerDied","Data":"5a2f0c8f29c3f8f6a369e31633240e04fe9b1c7f5228c5697f911a3d1d71431d"}
Nov 02 15:30:19.336553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.336482    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"
Nov 02 15:30:19.358740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.358713    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"]
Nov 02 15:30:19.413296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.411189    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"]
Nov 02 15:30:19.419146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.417777    7019 scope.go:122] "RemoveContainer" containerID="1de01af613970487622a67b20d33aed21bada2c5d1f275bcd1cf278742b5ea25"
Nov 02 15:30:19.440180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.440156    7019 scope.go:122] "RemoveContainer" containerID="f05a9eb32a0ab320a5a0c688f70f6b007c764ad9d764134dd0dec0e3f0ef2c70"
Nov 02 15:30:19.461996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.461780    7019 scope.go:122] "RemoveContainer" containerID="e567ad8e6966115a449183873d83225ee0ecb2505abb3954d850813b05f41425"
Nov 02 15:30:19.489892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.489870    7019 scope.go:122] "RemoveContainer" containerID="e259732248977141c706642252e6619addfc417c46af92952249127affd6bab2"
Nov 02 15:30:19.518765 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.518740    7019 scope.go:122] "RemoveContainer" containerID="13f081b7dd60a507c7019006fd58839b7647740a424c493bbfc231d781431cf9"
Nov 02 15:30:19.578006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.577305    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"]
Nov 02 15:30:19.584886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.584863    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893890-r8fp4"]
Nov 02 15:30:19.587571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.587496    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:19.660052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.660021    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"]
Nov 02 15:30:19.664984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.664955    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-78d075-29893890-8v72c"]
Nov 02 15:30:19.673872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.673842    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:19.694047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.693973    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-kube-api-access-pqj7j\" (UniqueName: \"kubernetes.io/projected/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-kube-api-access-pqj7j\") pod \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") "
Nov 02 15:30:19.694262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.694170    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-maas-api-service-ca\") pod \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") "
Nov 02 15:30:19.694541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.694320    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-maas-api-service-ca\") pod \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\" (UID: \"4fd3b658-cf74-49ee-b1d6-75141abc0dd4\") "
Nov 02 15:30:19.694541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.694366    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-kube-api-access-s2qwb\" (UniqueName: \"kubernetes.io/projected/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-kube-api-access-s2qwb\") pod \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\" (UID: \"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd\") "
Nov 02 15:30:19.694779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.694570    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-maas-api-service-ca" pod "efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" (UID: "efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:19.694850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.694768    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-maas-api-service-ca" pod "4fd3b658-cf74-49ee-b1d6-75141abc0dd4" (UID: "4fd3b658-cf74-49ee-b1d6-75141abc0dd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:19.695189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.695166    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.695276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.695195    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.696769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.696741    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-kube-api-access-pqj7j" pod "4fd3b658-cf74-49ee-b1d6-75141abc0dd4" (UID: "4fd3b658-cf74-49ee-b1d6-75141abc0dd4"). InnerVolumeSpecName "kube-api-access-pqj7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:19.696859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.696773    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-kube-api-access-s2qwb" pod "efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" (UID: "efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd"). InnerVolumeSpecName "kube-api-access-s2qwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:19.750815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.750783    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"]
Nov 02 15:30:19.755062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.755036    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2f8b7c-29893890-4grbp"]
Nov 02 15:30:19.794480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.794298    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:19.799007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.796768    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2qwb\" (UniqueName: \"kubernetes.io/projected/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd-kube-api-access-s2qwb\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.799007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.797541    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqj7j\" (UniqueName: \"kubernetes.io/projected/4fd3b658-cf74-49ee-b1d6-75141abc0dd4-kube-api-access-pqj7j\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:19.856701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.856665    7019 scope.go:122] "RemoveContainer" containerID="6a0b455d244642c72110805d6b89e547e223de458be1532cd01a1d9c3a52d81e"
Nov 02 15:30:19.858127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.858064    7019 scope.go:122] "RemoveContainer" containerID="c9145d9680880f4230dbe8158ac2fd1e3c0df04ffa03f93643c26ae06f7eae42"
Nov 02 15:30:19.882319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.882281    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" path="/var/lib/kubelet/pods/5748e924-6bf8-41b2-b5f6-b9e681456dcb/volumes"
Nov 02 15:30:19.883669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.883071    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" path="/var/lib/kubelet/pods/a2a57c32-b7d5-440a-adbc-6161142b71db/volumes"
Nov 02 15:30:19.883811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:19.883703    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" path="/var/lib/kubelet/pods/b9cbf4ea-ed62-4534-9619-4e3c2b04d271/volumes"
Nov 02 15:30:20.050093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.050059    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"]
Nov 02 15:30:20.054359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.054331    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-396b8a5a-vdhds"]
Nov 02 15:30:20.352140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.352106    7019 generic.go:356] "Generic (PLEG): container finished" podID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerID="ad749adad369ab2b2cd3ded9483e5590736162e4de74b0cc94571f1604b9e800" exitCode=60
Nov 02 15:30:20.352559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.352159    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" event={"ID":"1eb97b1a-5e61-4668-a9c4-31769504ed43","Type":"ContainerDied","Data":"ad749adad369ab2b2cd3ded9483e5590736162e4de74b0cc94571f1604b9e800"}
Nov 02 15:30:20.352559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.352227    7019 scope.go:122] "RemoveContainer" containerID="c8207278dff95be2c45346165447afe293e218cb56e195e875b1d0afa72f6d26"
Nov 02 15:30:20.352714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.352685    7019 scope.go:122] "RemoveContainer" containerID="ad749adad369ab2b2cd3ded9483e5590736162e4de74b0cc94571f1604b9e800"
Nov 02 15:30:20.353116 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:20.353091    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq_odh-ai-gateway-infra(1eb97b1a-5e61-4668-a9c4-31769504ed43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43"
Nov 02 15:30:20.354609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.354591    7019 generic.go:356] "Generic (PLEG): container finished" podID="55911434-6f57-4840-a09a-a67befd78f30" containerID="a54fa2c8d9c6af493b1e8d2ad466879ae56a009262d13a6802a92df467e92ee1" exitCode=60
Nov 02 15:30:20.354743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.354690    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" event={"ID":"55911434-6f57-4840-a09a-a67befd78f30","Type":"ContainerDied","Data":"a54fa2c8d9c6af493b1e8d2ad466879ae56a009262d13a6802a92df467e92ee1"}
Nov 02 15:30:20.355140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.355121    7019 scope.go:122] "RemoveContainer" containerID="a54fa2c8d9c6af493b1e8d2ad466879ae56a009262d13a6802a92df467e92ee1"
Nov 02 15:30:20.355524 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:20.355502    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8_odh-ai-gateway-infra(55911434-6f57-4840-a09a-a67befd78f30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" podUID="55911434-6f57-4840-a09a-a67befd78f30"
Nov 02 15:30:20.356799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.356780    7019 generic.go:356] "Generic (PLEG): container finished" podID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerID="46ae4019b415260b803be4bd9df2b99b70ffc4f4260a6206467555560e7ef9d8" exitCode=60
Nov 02 15:30:20.356880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.356852    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" event={"ID":"70b86377-400c-4ec3-8444-5e5c62e01fae","Type":"ContainerDied","Data":"46ae4019b415260b803be4bd9df2b99b70ffc4f4260a6206467555560e7ef9d8"}
Nov 02 15:30:20.357212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.357195    7019 scope.go:122] "RemoveContainer" containerID="46ae4019b415260b803be4bd9df2b99b70ffc4f4260a6206467555560e7ef9d8"
Nov 02 15:30:20.357562 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:20.357542    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc_odh-ai-gateway-infra(70b86377-400c-4ec3-8444-5e5c62e01fae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae"
Nov 02 15:30:20.360052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.360031    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg" event={"ID":"efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd","Type":"ContainerDied","Data":"cabddd835ee45230000f8d6d62266819de65e8f937eeb3cf273b7afc7b4465de"}
Nov 02 15:30:20.360052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.360043    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"
Nov 02 15:30:20.361268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.361251    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8" event={"ID":"4fd3b658-cf74-49ee-b1d6-75141abc0dd4","Type":"ContainerDied","Data":"8c75e9e6a2eb56cd76455ace1da95e968accea3f5bff6f6543a657854aea42e0"}
Nov 02 15:30:20.361339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.361328    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"
Nov 02 15:30:20.363461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.363438    7019 generic.go:356] "Generic (PLEG): container finished" podID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerID="136814b5c5f165fbc93d5f58d8baac9c7efbaf0819c2fc523e699190d99119eb" exitCode=60
Nov 02 15:30:20.363559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.363514    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" event={"ID":"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85","Type":"ContainerDied","Data":"136814b5c5f165fbc93d5f58d8baac9c7efbaf0819c2fc523e699190d99119eb"}
Nov 02 15:30:20.363938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.363919    7019 scope.go:122] "RemoveContainer" containerID="136814b5c5f165fbc93d5f58d8baac9c7efbaf0819c2fc523e699190d99119eb"
Nov 02 15:30:20.364226 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:20.364199    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f_odh-ai-gateway-infra(c8eb9b49-3ae4-4687-a406-a4c35d1dfd85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85"
Nov 02 15:30:20.421494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.421465    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"]
Nov 02 15:30:20.432975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.432953    7019 scope.go:122] "RemoveContainer" containerID="362892f1d5757928507e7826b8adc2242c319bd04207385b6cb7c07f4f215d63"
Nov 02 15:30:20.451811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.451782    7019 scope.go:122] "RemoveContainer" containerID="0ab574e8ef66e00a85c8838de8fb04ca1d6ecc205a258480a507f5d537e5cc72"
Nov 02 15:30:20.469945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.469921    7019 scope.go:122] "RemoveContainer" containerID="2bd3f5b242b8b5045e6f07bd7a029712eee9a2c396c36e488ffe3873640b3621"
Nov 02 15:30:20.485010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.484985    7019 scope.go:122] "RemoveContainer" containerID="dbfb48a9e76a12ead2527447d3ee01d9a09d06fc957569b7da9ca1545ac21c20"
Nov 02 15:30:20.489761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.489730    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"]
Nov 02 15:30:20.512924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.512891    7019 scope.go:122] "RemoveContainer" containerID="59f96dade6af768980077b7b1656fb311b0305b88d6befddce77fd87a710722f"
Nov 02 15:30:20.553218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.553183    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"]
Nov 02 15:30:20.646698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.643546    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:20.723333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.721691    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ca184a-7c3f-43fb-acb6-dce5d342b804-kube-api-access-ffn4l\" (UniqueName: \"kubernetes.io/projected/63ca184a-7c3f-43fb-acb6-dce5d342b804-kube-api-access-ffn4l\") pod \"63ca184a-7c3f-43fb-acb6-dce5d342b804\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") "
Nov 02 15:30:20.723333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.721774    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ca184a-7c3f-43fb-acb6-dce5d342b804-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ca184a-7c3f-43fb-acb6-dce5d342b804-maas-api-service-ca\") pod \"63ca184a-7c3f-43fb-acb6-dce5d342b804\" (UID: \"63ca184a-7c3f-43fb-acb6-dce5d342b804\") "
Nov 02 15:30:20.723333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.722660    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ca184a-7c3f-43fb-acb6-dce5d342b804-maas-api-service-ca" pod "63ca184a-7c3f-43fb-acb6-dce5d342b804" (UID: "63ca184a-7c3f-43fb-acb6-dce5d342b804"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.725910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.725889    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ca184a-7c3f-43fb-acb6-dce5d342b804-kube-api-access-ffn4l" pod "63ca184a-7c3f-43fb-acb6-dce5d342b804" (UID: "63ca184a-7c3f-43fb-acb6-dce5d342b804"). InnerVolumeSpecName "kube-api-access-ffn4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.771750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.771710    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:20.787633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.787553    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"]
Nov 02 15:30:20.793078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.793044    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893890-gxrbg"]
Nov 02 15:30:20.824188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.824150    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c84259f-d067-478d-ac0d-4c6087b95a2d-kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/3c84259f-d067-478d-ac0d-4c6087b95a2d-kube-api-access-2bgkr\") pod \"3c84259f-d067-478d-ac0d-4c6087b95a2d\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") "
Nov 02 15:30:20.824337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.824259    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c84259f-d067-478d-ac0d-4c6087b95a2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c84259f-d067-478d-ac0d-4c6087b95a2d-maas-api-service-ca\") pod \"3c84259f-d067-478d-ac0d-4c6087b95a2d\" (UID: \"3c84259f-d067-478d-ac0d-4c6087b95a2d\") "
Nov 02 15:30:20.825297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.825269    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffn4l\" (UniqueName: \"kubernetes.io/projected/63ca184a-7c3f-43fb-acb6-dce5d342b804-kube-api-access-ffn4l\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.825398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.825300    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63ca184a-7c3f-43fb-acb6-dce5d342b804-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.825674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.825635    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c84259f-d067-478d-ac0d-4c6087b95a2d-maas-api-service-ca" pod "3c84259f-d067-478d-ac0d-4c6087b95a2d" (UID: "3c84259f-d067-478d-ac0d-4c6087b95a2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.829219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.829115    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c84259f-d067-478d-ac0d-4c6087b95a2d-kube-api-access-2bgkr" pod "3c84259f-d067-478d-ac0d-4c6087b95a2d" (UID: "3c84259f-d067-478d-ac0d-4c6087b95a2d"). InnerVolumeSpecName "kube-api-access-2bgkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:20.835523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.835497    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:20.863952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.863928    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"]
Nov 02 15:30:20.867912 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.867886    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-f975ed-29893890-6nlx8"]
Nov 02 15:30:20.926319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.926284    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c69419-8ae3-488b-ae72-137540b1afc8-kube-api-access-frdjn\" (UniqueName: \"kubernetes.io/projected/91c69419-8ae3-488b-ae72-137540b1afc8-kube-api-access-frdjn\") pod \"91c69419-8ae3-488b-ae72-137540b1afc8\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") "
Nov 02 15:30:20.926523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.926505    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c69419-8ae3-488b-ae72-137540b1afc8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c69419-8ae3-488b-ae72-137540b1afc8-maas-api-service-ca\") pod \"91c69419-8ae3-488b-ae72-137540b1afc8\" (UID: \"91c69419-8ae3-488b-ae72-137540b1afc8\") "
Nov 02 15:30:20.926963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.926916    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c69419-8ae3-488b-ae72-137540b1afc8-maas-api-service-ca" pod "91c69419-8ae3-488b-ae72-137540b1afc8" (UID: "91c69419-8ae3-488b-ae72-137540b1afc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:20.927679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.927658    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bgkr\" (UniqueName: \"kubernetes.io/projected/3c84259f-d067-478d-ac0d-4c6087b95a2d-kube-api-access-2bgkr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.927909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.927882    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c84259f-d067-478d-ac0d-4c6087b95a2d-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.927999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.927909    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c69419-8ae3-488b-ae72-137540b1afc8-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:20.928856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:20.928828    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c69419-8ae3-488b-ae72-137540b1afc8-kube-api-access-frdjn" pod "91c69419-8ae3-488b-ae72-137540b1afc8" (UID: "91c69419-8ae3-488b-ae72-137540b1afc8"). InnerVolumeSpecName "kube-api-access-frdjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.029932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.029897    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdjn\" (UniqueName: \"kubernetes.io/projected/91c69419-8ae3-488b-ae72-137540b1afc8-kube-api-access-frdjn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.382938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.382904    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"
Nov 02 15:30:21.383399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.382932    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd" event={"ID":"63ca184a-7c3f-43fb-acb6-dce5d342b804","Type":"ContainerDied","Data":"c9885007de6f74a003361bc0c17d8c6ba1f8a98cdff69c16b5f984efaef4ddf9"}
Nov 02 15:30:21.383399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.383020    7019 scope.go:122] "RemoveContainer" containerID="1b0287f8cb263e4f0376dae934c89a71611a483941fd32d87c19adccbf983f53"
Nov 02 15:30:21.386339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.386316    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"
Nov 02 15:30:21.386339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.386327    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f" event={"ID":"3c84259f-d067-478d-ac0d-4c6087b95a2d","Type":"ContainerDied","Data":"8e083084bdf98217f078a5de769129ff59df51df6cf20f09d413ba3de2537988"}
Nov 02 15:30:21.388366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.388280    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855" event={"ID":"91c69419-8ae3-488b-ae72-137540b1afc8","Type":"ContainerDied","Data":"bdb7458801dbd7b9a9d559b2172e38327447bed7c6fd8d999abf7b6140eb11d5"}
Nov 02 15:30:21.388366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.388296    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"
Nov 02 15:30:21.418001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.417949    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"]
Nov 02 15:30:21.421326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.421293    7019 generic.go:356] "Generic (PLEG): container finished" podID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerID="0cffd7bf8995bfe12c69088ea9a0743120e9d4db154059f85899a7a99553fcdd" exitCode=60
Nov 02 15:30:21.421571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.421428    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" event={"ID":"4a739bb4-d8f5-4b46-a635-d486b4d7e314","Type":"ContainerDied","Data":"0cffd7bf8995bfe12c69088ea9a0743120e9d4db154059f85899a7a99553fcdd"}
Nov 02 15:30:21.422442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.422415    7019 scope.go:122] "RemoveContainer" containerID="0cffd7bf8995bfe12c69088ea9a0743120e9d4db154059f85899a7a99553fcdd"
Nov 02 15:30:21.422985 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:21.422961    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn_odh-ai-gateway-infra(4a739bb4-d8f5-4b46-a635-d486b4d7e314)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314"
Nov 02 15:30:21.424120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.424097    7019 scope.go:122] "RemoveContainer" containerID="2cc056a75b1aa275c90a77681fd7c9aeaf5a5f74857c67d2bc015f084303c054"
Nov 02 15:30:21.437070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.437037    7019 generic.go:356] "Generic (PLEG): container finished" podID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerID="128eeec15c12feb45ebbb9003d481b3ebda5a555725f42b7dc5b427348ef9edc" exitCode=60
Nov 02 15:30:21.437182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.437105    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" event={"ID":"921e240b-8fb7-4ea1-baf4-e8e539ade6ef","Type":"ContainerDied","Data":"128eeec15c12feb45ebbb9003d481b3ebda5a555725f42b7dc5b427348ef9edc"}
Nov 02 15:30:21.437569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.437547    7019 scope.go:122] "RemoveContainer" containerID="128eeec15c12feb45ebbb9003d481b3ebda5a555725f42b7dc5b427348ef9edc"
Nov 02 15:30:21.437974 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:21.437948    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b_odh-ai-gateway-infra(921e240b-8fb7-4ea1-baf4-e8e539ade6ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef"
Nov 02 15:30:21.439181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.439164    7019 scope.go:122] "RemoveContainer" containerID="916f95c3139d4465bd2a0dcc7b988649c323557d61639e78811f5a15867757ed"
Nov 02 15:30:21.450883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.450793    7019 scope.go:122] "RemoveContainer" containerID="6a0b455d244642c72110805d6b89e547e223de458be1532cd01a1d9c3a52d81e"
Nov 02 15:30:21.464041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.464009    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"]
Nov 02 15:30:21.467124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.467038    7019 scope.go:122] "RemoveContainer" containerID="c9145d9680880f4230dbe8158ac2fd1e3c0df04ffa03f93643c26ae06f7eae42"
Nov 02 15:30:21.523109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.523063    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"]
Nov 02 15:30:21.562726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.559207    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:21.581940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.579184    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"]
Nov 02 15:30:21.585156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.585115    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-5113c2-29893890-5whgd"]
Nov 02 15:30:21.614247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.614213    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"]
Nov 02 15:30:21.696679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.695581    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"]
Nov 02 15:30:21.702576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.702547    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-1cc1b2-29893890-2mp6f"]
Nov 02 15:30:21.730187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.730107    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:21.751014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.750979    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"]
Nov 02 15:30:21.763269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.762149    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb97b1a-5e61-4668-a9c4-31769504ed43-kube-api-access-p79dv\" (UniqueName: \"kubernetes.io/projected/1eb97b1a-5e61-4668-a9c4-31769504ed43-kube-api-access-p79dv\") pod \"1eb97b1a-5e61-4668-a9c4-31769504ed43\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") "
Nov 02 15:30:21.763269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.762265    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb97b1a-5e61-4668-a9c4-31769504ed43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb97b1a-5e61-4668-a9c4-31769504ed43-maas-api-service-ca\") pod \"1eb97b1a-5e61-4668-a9c4-31769504ed43\" (UID: \"1eb97b1a-5e61-4668-a9c4-31769504ed43\") "
Nov 02 15:30:21.764126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.763840    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-80ec03-29893890-zr855"]
Nov 02 15:30:21.764420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.764378    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb97b1a-5e61-4668-a9c4-31769504ed43-maas-api-service-ca" pod "1eb97b1a-5e61-4668-a9c4-31769504ed43" (UID: "1eb97b1a-5e61-4668-a9c4-31769504ed43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.798689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.798618    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb97b1a-5e61-4668-a9c4-31769504ed43-kube-api-access-p79dv" pod "1eb97b1a-5e61-4668-a9c4-31769504ed43" (UID: "1eb97b1a-5e61-4668-a9c4-31769504ed43"). InnerVolumeSpecName "kube-api-access-p79dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.835763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.835729    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:21.865106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.864779    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b86377-400c-4ec3-8444-5e5c62e01fae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b86377-400c-4ec3-8444-5e5c62e01fae-maas-api-service-ca\") pod \"70b86377-400c-4ec3-8444-5e5c62e01fae\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") "
Nov 02 15:30:21.865106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.864988    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b86377-400c-4ec3-8444-5e5c62e01fae-kube-api-access-wdl68\" (UniqueName: \"kubernetes.io/projected/70b86377-400c-4ec3-8444-5e5c62e01fae-kube-api-access-wdl68\") pod \"70b86377-400c-4ec3-8444-5e5c62e01fae\" (UID: \"70b86377-400c-4ec3-8444-5e5c62e01fae\") "
Nov 02 15:30:21.866131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.866105    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p79dv\" (UniqueName: \"kubernetes.io/projected/1eb97b1a-5e61-4668-a9c4-31769504ed43-kube-api-access-p79dv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.866567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.866250    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb97b1a-5e61-4668-a9c4-31769504ed43-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.869141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.868941    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b86377-400c-4ec3-8444-5e5c62e01fae-kube-api-access-wdl68" pod "70b86377-400c-4ec3-8444-5e5c62e01fae" (UID: "70b86377-400c-4ec3-8444-5e5c62e01fae"). InnerVolumeSpecName "kube-api-access-wdl68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:21.872560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.869400    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b86377-400c-4ec3-8444-5e5c62e01fae-maas-api-service-ca" pod "70b86377-400c-4ec3-8444-5e5c62e01fae" (UID: "70b86377-400c-4ec3-8444-5e5c62e01fae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.875560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.875483    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:21.877708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.877683    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c7cc07-5f41-4eb9-9fa3-facf540edbf1" path="/var/lib/kubelet/pods/28c7cc07-5f41-4eb9-9fa3-facf540edbf1/volumes"
Nov 02 15:30:21.878507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.878486    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" path="/var/lib/kubelet/pods/3c84259f-d067-478d-ac0d-4c6087b95a2d/volumes"
Nov 02 15:30:21.879236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.879216    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" path="/var/lib/kubelet/pods/4fd3b658-cf74-49ee-b1d6-75141abc0dd4/volumes"
Nov 02 15:30:21.880381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.880362    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" path="/var/lib/kubelet/pods/63ca184a-7c3f-43fb-acb6-dce5d342b804/volumes"
Nov 02 15:30:21.881044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.881025    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" path="/var/lib/kubelet/pods/91c69419-8ae3-488b-ae72-137540b1afc8/volumes"
Nov 02 15:30:21.881785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.881765    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" path="/var/lib/kubelet/pods/efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd/volumes"
Nov 02 15:30:21.939915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.939890    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:21.966901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.966857    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-maas-api-service-ca\") pod \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") "
Nov 02 15:30:21.968058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.967834    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-maas-api-service-ca" pod "c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" (UID: "c8eb9b49-3ae4-4687-a406-a4c35d1dfd85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:21.969118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.968885    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-kube-api-access-4tlmj\" (UniqueName: \"kubernetes.io/projected/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-kube-api-access-4tlmj\") pod \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\" (UID: \"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85\") "
Nov 02 15:30:21.972723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.972705    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b86377-400c-4ec3-8444-5e5c62e01fae-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.972853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.972836    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdl68\" (UniqueName: \"kubernetes.io/projected/70b86377-400c-4ec3-8444-5e5c62e01fae-kube-api-access-wdl68\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.972942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.972857    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:21.974630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:21.974595    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-kube-api-access-4tlmj" pod "c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" (UID: "c8eb9b49-3ae4-4687-a406-a4c35d1dfd85"). InnerVolumeSpecName "kube-api-access-4tlmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:22.073497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.073467    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55911434-6f57-4840-a09a-a67befd78f30-kube-api-access-wwj8k\" (UniqueName: \"kubernetes.io/projected/55911434-6f57-4840-a09a-a67befd78f30-kube-api-access-wwj8k\") pod \"55911434-6f57-4840-a09a-a67befd78f30\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") "
Nov 02 15:30:22.073738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.073659    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55911434-6f57-4840-a09a-a67befd78f30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55911434-6f57-4840-a09a-a67befd78f30-maas-api-service-ca\") pod \"55911434-6f57-4840-a09a-a67befd78f30\" (UID: \"55911434-6f57-4840-a09a-a67befd78f30\") "
Nov 02 15:30:22.074053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.074022    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55911434-6f57-4840-a09a-a67befd78f30-maas-api-service-ca" pod "55911434-6f57-4840-a09a-a67befd78f30" (UID: "55911434-6f57-4840-a09a-a67befd78f30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:22.074502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.074477    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55911434-6f57-4840-a09a-a67befd78f30-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.074680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.074633    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tlmj\" (UniqueName: \"kubernetes.io/projected/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85-kube-api-access-4tlmj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.076268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.076243    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55911434-6f57-4840-a09a-a67befd78f30-kube-api-access-wwj8k" pod "55911434-6f57-4840-a09a-a67befd78f30" (UID: "55911434-6f57-4840-a09a-a67befd78f30"). InnerVolumeSpecName "kube-api-access-wwj8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:22.176992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.176953    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwj8k\" (UniqueName: \"kubernetes.io/projected/55911434-6f57-4840-a09a-a67befd78f30-kube-api-access-wwj8k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.449898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.449869    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"
Nov 02 15:30:22.450335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.449876    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f" event={"ID":"c8eb9b49-3ae4-4687-a406-a4c35d1dfd85","Type":"ContainerDied","Data":"79cc2ae5efc32377bda36e18ec218694b334b9d83c40b42f2a5452526571228b"}
Nov 02 15:30:22.450335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.450010    7019 scope.go:122] "RemoveContainer" containerID="136814b5c5f165fbc93d5f58d8baac9c7efbaf0819c2fc523e699190d99119eb"
Nov 02 15:30:22.455059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.455030    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"
Nov 02 15:30:22.455059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.455050    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq" event={"ID":"1eb97b1a-5e61-4668-a9c4-31769504ed43","Type":"ContainerDied","Data":"2120521ac44c1eef68309ad0b1a30b0b6d44a2af6cde5afcdd76f3bc2164b7f1"}
Nov 02 15:30:22.458583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.458556    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"
Nov 02 15:30:22.458735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.458577    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8" event={"ID":"55911434-6f57-4840-a09a-a67befd78f30","Type":"ContainerDied","Data":"57a6735b683834a2a7355773df6e3bb992651ef916b7aef317751a8e41dd1df5"}
Nov 02 15:30:22.460222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.460205    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"
Nov 02 15:30:22.460222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.460219    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc" event={"ID":"70b86377-400c-4ec3-8444-5e5c62e01fae","Type":"ContainerDied","Data":"c2de8317256a7c9d45c1ef1af733b58ffeb6e10d645372e009ad68ed87d68d13"}
Nov 02 15:30:22.464128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.464083    7019 scope.go:122] "RemoveContainer" containerID="ad749adad369ab2b2cd3ded9483e5590736162e4de74b0cc94571f1604b9e800"
Nov 02 15:30:22.478311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.478289    7019 scope.go:122] "RemoveContainer" containerID="a54fa2c8d9c6af493b1e8d2ad466879ae56a009262d13a6802a92df467e92ee1"
Nov 02 15:30:22.489378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.489351    7019 scope.go:122] "RemoveContainer" containerID="46ae4019b415260b803be4bd9df2b99b70ffc4f4260a6206467555560e7ef9d8"
Nov 02 15:30:22.517679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.517628    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"]
Nov 02 15:30:22.530909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.530878    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"]
Nov 02 15:30:22.534699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.534639    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a03e84-29893890-v8plq"]
Nov 02 15:30:22.574955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.574624    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"]
Nov 02 15:30:22.580758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.580728    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-2abeaa-29893890-84vjc"]
Nov 02 15:30:22.621857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.620968    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"]
Nov 02 15:30:22.631272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.631235    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-3daba5-29893890-ghd6f"]
Nov 02 15:30:22.669746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.669709    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"]
Nov 02 15:30:22.719592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.719531    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"]
Nov 02 15:30:22.723696 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:22.723614    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:30:22.723828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.723757    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:22.725927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.725904    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-cce791-29893890-w74d8"]
Nov 02 15:30:22.789861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.787820    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-maas-api-service-ca\") pod \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") "
Nov 02 15:30:22.789861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.787922    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-kube-api-access-njm9t\" (UniqueName: \"kubernetes.io/projected/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-kube-api-access-njm9t\") pod \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\" (UID: \"921e240b-8fb7-4ea1-baf4-e8e539ade6ef\") "
Nov 02 15:30:22.789861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.788488    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-maas-api-service-ca" pod "921e240b-8fb7-4ea1-baf4-e8e539ade6ef" (UID: "921e240b-8fb7-4ea1-baf4-e8e539ade6ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:22.789861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.788762    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.792777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.791296    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-kube-api-access-njm9t" pod "921e240b-8fb7-4ea1-baf4-e8e539ade6ef" (UID: "921e240b-8fb7-4ea1-baf4-e8e539ade6ef"). InnerVolumeSpecName "kube-api-access-njm9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:22.855978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.855954    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:22.890174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.890138    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njm9t\" (UniqueName: \"kubernetes.io/projected/921e240b-8fb7-4ea1-baf4-e8e539ade6ef-kube-api-access-njm9t\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.990758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.990718    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a739bb4-d8f5-4b46-a635-d486b4d7e314-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a739bb4-d8f5-4b46-a635-d486b4d7e314-maas-api-service-ca\") pod \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") "
Nov 02 15:30:22.991121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.991096    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a739bb4-d8f5-4b46-a635-d486b4d7e314-kube-api-access-w62lc\" (UniqueName: \"kubernetes.io/projected/4a739bb4-d8f5-4b46-a635-d486b4d7e314-kube-api-access-w62lc\") pod \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\" (UID: \"4a739bb4-d8f5-4b46-a635-d486b4d7e314\") "
Nov 02 15:30:22.991255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.991214    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a739bb4-d8f5-4b46-a635-d486b4d7e314-maas-api-service-ca" pod "4a739bb4-d8f5-4b46-a635-d486b4d7e314" (UID: "4a739bb4-d8f5-4b46-a635-d486b4d7e314"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:30:22.992484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.992261    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a739bb4-d8f5-4b46-a635-d486b4d7e314-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:22.994388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:22.994365    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a739bb4-d8f5-4b46-a635-d486b4d7e314-kube-api-access-w62lc" pod "4a739bb4-d8f5-4b46-a635-d486b4d7e314" (UID: "4a739bb4-d8f5-4b46-a635-d486b4d7e314"). InnerVolumeSpecName "kube-api-access-w62lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:30:23.094328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.094290    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w62lc\" (UniqueName: \"kubernetes.io/projected/4a739bb4-d8f5-4b46-a635-d486b4d7e314-kube-api-access-w62lc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:30:23.285558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.285460    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:23.470971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.470943    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"
Nov 02 15:30:23.470971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.470955    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn" event={"ID":"4a739bb4-d8f5-4b46-a635-d486b4d7e314","Type":"ContainerDied","Data":"d131efd804b4c484248d96b088b845586223f0e58b82565af19eb9df1fd8d4bc"}
Nov 02 15:30:23.471514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.471028    7019 scope.go:122] "RemoveContainer" containerID="0cffd7bf8995bfe12c69088ea9a0743120e9d4db154059f85899a7a99553fcdd"
Nov 02 15:30:23.475187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.475164    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"
Nov 02 15:30:23.475326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.475197    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b" event={"ID":"921e240b-8fb7-4ea1-baf4-e8e539ade6ef","Type":"ContainerDied","Data":"e001f9753fb3cf7ca6856a5c754d0e1f381588e95a37d5bde60da3cc9f0a5648"}
Nov 02 15:30:23.484761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.484565    7019 scope.go:122] "RemoveContainer" containerID="128eeec15c12feb45ebbb9003d481b3ebda5a555725f42b7dc5b427348ef9edc"
Nov 02 15:30:23.528143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.528112    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"]
Nov 02 15:30:23.531548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.531522    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0a7163-29893890-wvr8b"]
Nov 02 15:30:23.579696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.579597    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"]
Nov 02 15:30:23.589965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.589936    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d35a86-29893890-4v6mn"]
Nov 02 15:30:23.864624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.864593    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" path="/var/lib/kubelet/pods/1eb97b1a-5e61-4668-a9c4-31769504ed43/volumes"
Nov 02 15:30:23.865139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.865123    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" path="/var/lib/kubelet/pods/4a739bb4-d8f5-4b46-a635-d486b4d7e314/volumes"
Nov 02 15:30:23.865527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.865514    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55911434-6f57-4840-a09a-a67befd78f30" path="/var/lib/kubelet/pods/55911434-6f57-4840-a09a-a67befd78f30/volumes"
Nov 02 15:30:23.866287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.866271    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" path="/var/lib/kubelet/pods/70b86377-400c-4ec3-8444-5e5c62e01fae/volumes"
Nov 02 15:30:23.866695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.866681    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" path="/var/lib/kubelet/pods/921e240b-8fb7-4ea1-baf4-e8e539ade6ef/volumes"
Nov 02 15:30:23.867082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:23.867069    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" path="/var/lib/kubelet/pods/c8eb9b49-3ae4-4687-a406-a4c35d1dfd85/volumes"
Nov 02 15:30:24.873127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:24.873087    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:26.418432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:26.418215    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:28.230288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:28.230242    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:29.585234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:29.574926    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:30:30.149011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:30.148971    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:31.818904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.818860    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"]
Nov 02 15:30:31.820822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.820794    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:31.821006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.820992    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:31.821104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.821094    7019 cpu_manager.go:414] "containerMap: removing container" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.821188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.821176    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.821275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.821266    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:31.821349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.821340    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:31.821425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.821416    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:31.821510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.821496    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:31.821600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.821590    7019 cpu_manager.go:414] "containerMap: removing container" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.850458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.850423    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.850716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.850700    7019 cpu_manager.go:414] "containerMap: removing container" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.850818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.850806    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.850910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.850901    7019 cpu_manager.go:414] "containerMap: removing container" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.850991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.850981    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.851082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851072    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:31.851162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851152    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:31.851252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851242    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:31.851337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851327    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:31.851420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851409    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:31.851531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851515    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:31.851636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851624    7019 cpu_manager.go:414] "containerMap: removing container" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.851754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851741    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.851844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851833    7019 cpu_manager.go:414] "containerMap: removing container" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.851927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.851916    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.852058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852046    7019 cpu_manager.go:414] "containerMap: removing container" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:31.852128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852120    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:31.852181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852175    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:31.852228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852221    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:31.852287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852281    7019 cpu_manager.go:414] "containerMap: removing container" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.852335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852328    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.852388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852381    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:31.852435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852429    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:31.852486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852480    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:31.852532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852526    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:31.852588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852582    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:31.852638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852631    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:31.852714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852702    7019 cpu_manager.go:414] "containerMap: removing container" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.852775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852717    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.852775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852729    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:31.852775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852737    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:31.852775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852747    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:31.852775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852754    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:31.852775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852764    7019 cpu_manager.go:414] "containerMap: removing container" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:31.852775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852772    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852783    7019 cpu_manager.go:414] "containerMap: removing container" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852790    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852806    7019 cpu_manager.go:414] "containerMap: removing container" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852814    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852823    7019 cpu_manager.go:414] "containerMap: removing container" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852829    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852850    7019 cpu_manager.go:414] "containerMap: removing container" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852856    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852867    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852874    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852886    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852893    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852922    7019 cpu_manager.go:414] "containerMap: removing container" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852930    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852949    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852956    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852969    7019 cpu_manager.go:414] "containerMap: removing container" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.852976    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853003    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853011    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:31.853217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853226    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853244    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853253    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853263    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853271    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853280    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853289    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853297    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853309    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853317    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853329    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853337    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853344    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853352    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853364    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853374    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853383    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853392    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853400    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853411    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853422    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853431    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853441    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853449    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853459    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853467    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853476    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853486    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853552    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853563    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:31.854423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.853571    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:31.862556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.862531    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"]
Nov 02 15:30:31.862788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.862767    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:31.891813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.891552    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-b955c5-serving-cert\""
Nov 02 15:30:31.936601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:31.936562    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:32.083962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.078475    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.083962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.078563    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.083962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.078685    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn9rm\" (UniqueName: \"kubernetes.io/projected/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-kube-api-access-sn9rm\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.083962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.078956    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.118318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.118279    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:32.182162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.182119    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.183595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.183567    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.184215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.184191    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.197773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.197743    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn9rm\" (UniqueName: \"kubernetes.io/projected/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-kube-api-access-sn9rm\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.198537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.196583    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.216585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.216551    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.243884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.243849    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.290674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.285427    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn9rm\" (UniqueName: \"kubernetes.io/projected/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-kube-api-access-sn9rm\") pod \"maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:32.556076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:32.556036    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:33.242680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:33.234619    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"]
Nov 02 15:30:33.327671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:33.325099    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:33.763895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:33.763857    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf" event={"ID":"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8","Type":"ContainerStarted","Data":"b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56"}
Nov 02 15:30:33.764100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:33.763920    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf" event={"ID":"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8","Type":"ContainerStarted","Data":"3d92efa6685b768092e2af8304ee3a2be8ccf5713ac6fb948814dc35eadad3a2"}
Nov 02 15:30:33.768711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:33.764721    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:33.820306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:33.820247    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf" podStartSLOduration=2.354927027 podStartE2EDuration="2.820227341s" totalImagesPullingTime="465.300314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:33.814887562 +0000 UTC m=+2028.068841187" watchObservedRunningTime="2026-11-02 15:30:33.820227341 +0000 UTC m=+2028.074180966"
Nov 02 15:30:33.902727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:33.898267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:34.366521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.366480    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"]
Nov 02 15:30:34.368318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.368288    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:34.368485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.368471    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd3b658-cf74-49ee-b1d6-75141abc0dd4" containerName="cleanup"
Nov 02 15:30:34.368618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.368603    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:34.368752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.368737    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:34.368859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.368846    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:34.368960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.368948    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:34.369052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369040    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:34.369147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369135    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:34.369240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369227    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:34.369323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369313    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:34.369422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369410    7019 cpu_manager.go:414] "containerMap: removing container" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:34.369500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369489    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:34.369581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369571    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:34.369681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369668    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a57c32-b7d5-440a-adbc-6161142b71db" containerName="cleanup"
Nov 02 15:30:34.369794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369782    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:34.369882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369871    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c84259f-d067-478d-ac0d-4c6087b95a2d" containerName="cleanup"
Nov 02 15:30:34.369998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.369988    7019 cpu_manager.go:414] "containerMap: removing container" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:34.370081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370070    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:34.370178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370168    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:34.370259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370249    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:34.370519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370504    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8eb9b49-3ae4-4687-a406-a4c35d1dfd85" containerName="cleanup"
Nov 02 15:30:34.370625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370614    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb97b1a-5e61-4668-a9c4-31769504ed43" containerName="cleanup"
Nov 02 15:30:34.370819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370804    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="921e240b-8fb7-4ea1-baf4-e8e539ade6ef" containerName="cleanup"
Nov 02 15:30:34.370911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370901    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cbf4ea-ed62-4534-9619-4e3c2b04d271" containerName="cleanup"
Nov 02 15:30:34.371006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.370996    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b86377-400c-4ec3-8444-5e5c62e01fae" containerName="cleanup"
Nov 02 15:30:34.371092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.371082    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ca184a-7c3f-43fb-acb6-dce5d342b804" containerName="cleanup"
Nov 02 15:30:34.371179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.371169    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbcfbcb-ea31-4d9c-82f6-111a80a7b0cd" containerName="cleanup"
Nov 02 15:30:34.371257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.371248    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="55911434-6f57-4840-a09a-a67befd78f30" containerName="cleanup"
Nov 02 15:30:34.371350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.371339    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a739bb4-d8f5-4b46-a635-d486b4d7e314" containerName="cleanup"
Nov 02 15:30:34.371436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.371424    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5748e924-6bf8-41b2-b5f6-b9e681456dcb" containerName="cleanup"
Nov 02 15:30:34.371532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.371520    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c69419-8ae3-488b-ae72-137540b1afc8" containerName="cleanup"
Nov 02 15:30:34.401011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.400977    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.405322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.405289    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"]
Nov 02 15:30:34.421059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.421031    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-31c816\"/\"default-dockercfg-jvkhq\""
Nov 02 15:30:34.421222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.421158    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-31c816\"/\"kube-root-ca.crt\""
Nov 02 15:30:34.421222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.421210    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-31c816\"/\"openshift-service-ca.crt\""
Nov 02 15:30:34.433853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.433825    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-31c816\"/\"test-model-31c816-kserve-self-signed-certs\""
Nov 02 15:30:34.437169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.437136    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-home\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.437518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.437495    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tmp-dir\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.437782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.437763    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tls-certs\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.437948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.437930    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-model-cache\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.438070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.438053    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79sgc\" (UniqueName: \"kubernetes.io/projected/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-kube-api-access-79sgc\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.438239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.438224    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-dshm\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.486674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.486410    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:34.539786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.539737    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tls-certs\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.539829    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-model-cache\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.539862    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79sgc\" (UniqueName: \"kubernetes.io/projected/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-kube-api-access-79sgc\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.539945    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-dshm\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.540076    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-home\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.540223    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tmp-dir\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.540310    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-model-cache\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.540492    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-home\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.540588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.540549    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tmp-dir\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.542417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.542390    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-dshm\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.542660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.542626    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tls-certs\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.564454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.564419    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79sgc\" (UniqueName: \"kubernetes.io/projected/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-kube-api-access-79sgc\") pod \"test-model-31c816-kserve-6c6849585f-qxpjg\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") " pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:34.720150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:34.720026    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:35.286739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:35.285701    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"]
Nov 02 15:30:35.306675 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:30:35.306034    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71e23bdb_9879_4fe0_9de0_5f3a1b687c17.slice/crio-7227fea5470a2d98e36fbc932d5fe0afcfcf6f37b159b576246d5a58cb620c55 WatchSource:0}: Error finding container 7227fea5470a2d98e36fbc932d5fe0afcfcf6f37b159b576246d5a58cb620c55: Status 404 returned error can't find the container with id 7227fea5470a2d98e36fbc932d5fe0afcfcf6f37b159b576246d5a58cb620c55
Nov 02 15:30:35.691672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:35.691270    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:35.893670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:35.893435    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg" event={"ID":"71e23bdb-9879-4fe0-9de0-5f3a1b687c17","Type":"ContainerStarted","Data":"f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712"}
Nov 02 15:30:35.893670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:35.893489    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg" event={"ID":"71e23bdb-9879-4fe0-9de0-5f3a1b687c17","Type":"ContainerStarted","Data":"7227fea5470a2d98e36fbc932d5fe0afcfcf6f37b159b576246d5a58cb620c55"}
Nov 02 15:30:35.942552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:35.942419    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg" podStartSLOduration=1.9423983740000001 podStartE2EDuration="1.942398374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:35.939560554 +0000 UTC m=+2030.193514179" watchObservedRunningTime="2026-11-02 15:30:35.942398374 +0000 UTC m=+2030.196351999"
Nov 02 15:30:36.721776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:36.721737    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:37.382668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:37.369233    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:38.989792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:38.989753    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"]
Nov 02 15:30:38.990798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:38.990772    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:38.997163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:38.996289    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.026706    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-69ff88\"/\"kube-root-ca.crt\""
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.026838    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-69ff88\"/\"test-model-69ff88-kserve-self-signed-certs\""
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.026941    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-69ff88\"/\"default-dockercfg-gkzfh\""
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.027546    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-69ff88\"/\"openshift-service-ca.crt\""
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.029374    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-dshm\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.029417    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-model-cache\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.029464    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tls-certs\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.029675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.029592    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-home\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.034399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.032993    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"]
Nov 02 15:30:39.034399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.033940    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7lfn\" (UniqueName: \"kubernetes.io/projected/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-kube-api-access-m7lfn\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.034399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.034182    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tmp-dir\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.136172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.136132    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tmp-dir\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.136368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.136227    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-dshm\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.136368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.136261    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-model-cache\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.136368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.136308    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tls-certs\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.136673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.136421    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-home\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.136673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.136488    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7lfn\" (UniqueName: \"kubernetes.io/projected/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-kube-api-access-m7lfn\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.137278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.137249    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tmp-dir\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.138029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.137988    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-home\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.138127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.138056    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-model-cache\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.140017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.139970    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-dshm\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.140763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.140722    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tls-certs\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.151907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.151794    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:39.162272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.161950    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7lfn\" (UniqueName: \"kubernetes.io/projected/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-kube-api-access-m7lfn\") pod \"test-model-69ff88-kserve-967979dfc-hvv8v\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") " pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.334866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.333597    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:39.811079 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:30:39.811044    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ee2b4d6_d1f2_4577_9c94_419f89c037b2.slice/crio-46edefc9e9042764b9f33decb0de169bae76b855724b3d693e83ca667e8d2b11 WatchSource:0}: Error finding container 46edefc9e9042764b9f33decb0de169bae76b855724b3d693e83ca667e8d2b11: Status 404 returned error can't find the container with id 46edefc9e9042764b9f33decb0de169bae76b855724b3d693e83ca667e8d2b11
Nov 02 15:30:39.824705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.816636    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"]
Nov 02 15:30:39.877809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.877781    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:30:39.949145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.948741    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v" event={"ID":"1ee2b4d6-d1f2-4577-9c94-419f89c037b2","Type":"ContainerStarted","Data":"0c73119aa11fd528cada619a32c6ceeada8c33cf9ae33b4b6afc694705cd8d65"}
Nov 02 15:30:39.949145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:39.948795    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v" event={"ID":"1ee2b4d6-d1f2-4577-9c94-419f89c037b2","Type":"ContainerStarted","Data":"46edefc9e9042764b9f33decb0de169bae76b855724b3d693e83ca667e8d2b11"}
Nov 02 15:30:40.000460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:40.000407    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v" podStartSLOduration=2.000392546 podStartE2EDuration="2.000392546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:39.996979915 +0000 UTC m=+2034.250933540" watchObservedRunningTime="2026-11-02 15:30:40.000392546 +0000 UTC m=+2034.254346170"
Nov 02 15:30:40.334071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:40.334035    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:40.613532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:40.613503    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:42.266344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:42.266253    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:42.975527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:42.975488    7019 generic.go:356] "Generic (PLEG): container finished" podID="29329576-824e-44bb-a825-b93ab7391426" containerID="592225f8325dfea14ba8db1eced35d4fe12af35bbfd52a0b0db67c4cd5ee542f" exitCode=0
Nov 02 15:30:42.975751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:42.975549    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" event={"ID":"29329576-824e-44bb-a825-b93ab7391426","Type":"ContainerDied","Data":"592225f8325dfea14ba8db1eced35d4fe12af35bbfd52a0b0db67c4cd5ee542f"}
Nov 02 15:30:42.976151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:42.976133    7019 scope.go:122] "RemoveContainer" containerID="592225f8325dfea14ba8db1eced35d4fe12af35bbfd52a0b0db67c4cd5ee542f"
Nov 02 15:30:42.978191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:42.978153    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:42.978312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:42.978241    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:42.978312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:42.978293    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:30:43.897270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:43.897207    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:44.006062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:44.006024    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" event={"ID":"29329576-824e-44bb-a825-b93ab7391426","Type":"ContainerStarted","Data":"cf6f6954b12a933dcc30ccba3a8630386de1a57a9f0869fd0898a46ac0492821"}
Nov 02 15:30:44.034387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:44.034349    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:30:44.034679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:44.034625    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:30:44.034743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:44.034696    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 15:30:44.721276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:44.721226    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:44.734276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:44.734242    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:45.028886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:45.028802    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:30:45.034162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:45.034137    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:30:45.034257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:45.034189    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 15:30:45.535686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:45.535636    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:46.033939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:46.033909    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:30:46.034335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:46.033956    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 15:30:47.034341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:47.034303    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:30:47.034797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:47.034368    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 15:30:47.218343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:47.218302    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:47.963852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:47.963820    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:48.037861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:48.037819    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:30:48.038374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:48.037885    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 15:30:48.239006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:48.238910    7019 scope.go:122] "RemoveContainer" containerID="1c2ab8a02d8d77210531a12ef8ab5f2c2bb1fb7652039a0b48c73592137ee4e6"
Nov 02 15:30:48.926818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:48.926780    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:49.034702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:49.034182    7019 patch_prober.go:29] interesting pod/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 15:30:49.034702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:49.034239    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 15:30:49.173952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:49.172807    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:30:49.334492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:49.334412    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:49.347861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:49.347833    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:50.038781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:50.038750    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:30:50.061372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:50.061346    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:30:50.075699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:50.075638    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:30:50.590484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:50.590450    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:52.202473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:52.202435    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:52.707906 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:30:52.707862    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:30:53.747133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:53.747097    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:55.393899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:55.393859    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:56.883420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.883382    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"]
Nov 02 15:30:56.890437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.890408    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:56.916804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.916767    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"]
Nov 02 15:30:56.922009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.921980    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2e4ebc\"/\"openshift-service-ca.crt\""
Nov 02 15:30:56.922009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.921987    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2e4ebc\"/\"kube-root-ca.crt\""
Nov 02 15:30:56.922213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.922053    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2e4ebc\"/\"default-dockercfg-hscbn\""
Nov 02 15:30:56.922691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.922667    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2e4ebc\"/\"test-model-2e4ebc-kserve-self-signed-certs\""
Nov 02 15:30:56.950146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.950100    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6345111-1120-4afd-af75-733bae7aa495-tls-certs\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:56.950312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.950212    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdfm\" (UniqueName: \"kubernetes.io/projected/a6345111-1120-4afd-af75-733bae7aa495-kube-api-access-6zdfm\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:56.950387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.950354    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-dshm\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:56.950531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.950504    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-model-cache\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:56.950595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.950552    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-home\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:56.950696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.950673    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-tmp-dir\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:56.976548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:56.976517    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:57.052784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.052744    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6345111-1120-4afd-af75-733bae7aa495-tls-certs\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.052989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.052896    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdfm\" (UniqueName: \"kubernetes.io/projected/a6345111-1120-4afd-af75-733bae7aa495-kube-api-access-6zdfm\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.052989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.052955    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-dshm\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.053115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.053041    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-model-cache\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.053115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.053083    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-home\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.053233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.053144    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-tmp-dir\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.053542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.053516    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-model-cache\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.053611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.053527    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-home\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.053611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.053578    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-tmp-dir\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.055614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.055588    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6345111-1120-4afd-af75-733bae7aa495-tls-certs\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.055857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.055833    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-dshm\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.092726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.092685    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdfm\" (UniqueName: \"kubernetes.io/projected/a6345111-1120-4afd-af75-733bae7aa495-kube-api-access-6zdfm\") pod \"test-model-2e4ebc-kserve-55bb8f569c-z2htd\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") " pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.228305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.227931    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:57.269517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.269478    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:57.709396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:57.709366    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"]
Nov 02 15:30:58.135929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.135890    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd" event={"ID":"a6345111-1120-4afd-af75-733bae7aa495","Type":"ContainerStarted","Data":"b2f101466ac9e7a3002c99b9c46a973971ca395a6f75a17fcd8def3aabb84d5f"}
Nov 02 15:30:58.136422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.135947    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd" event={"ID":"a6345111-1120-4afd-af75-733bae7aa495","Type":"ContainerStarted","Data":"c04b96f31b2548152c530ccb7e28b4eec67fc564db955b6d9910970cafa25ea9"}
Nov 02 15:30:58.182268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.182215    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd" podStartSLOduration=2.1822012060000002 podStartE2EDuration="2.182201206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:58.178738509 +0000 UTC m=+2052.432692133" watchObservedRunningTime="2026-11-02 15:30:58.182201206 +0000 UTC m=+2052.436154830"
Nov 02 15:30:58.229616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.229580    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:30:58.793044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.792977    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:30:58.964178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.964144    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"]
Nov 02 15:30:58.967899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.967875    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:58.990063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.990033    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"]
Nov 02 15:30:59.004187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.998582    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:30:59.004187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:58.998637    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:30:59.096757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.096628    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsrs\" (UniqueName: \"kubernetes.io/projected/c4c9dce5-617c-4086-b13c-cd178b6bff07-kube-api-access-lmsrs\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.096945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.096768    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.096945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.096899    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.198264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.198208    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.198773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.198580    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsrs\" (UniqueName: \"kubernetes.io/projected/c4c9dce5-617c-4086-b13c-cd178b6bff07-kube-api-access-lmsrs\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.198773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.198674    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.199534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.199507    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.201161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.201138    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.227609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.227576    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsrs\" (UniqueName: \"kubernetes.io/projected/c4c9dce5-617c-4086-b13c-cd178b6bff07-kube-api-access-lmsrs\") pod \"maas-api-revoke-sub-a62e776e-8fjcn\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.286422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.286393    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:30:59.600583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:30:59.600544    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"]
Nov 02 15:31:00.166053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:00.166002    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn" event={"ID":"c4c9dce5-617c-4086-b13c-cd178b6bff07","Type":"ContainerStarted","Data":"6714f2e9851c84253a49bcc7e46a956fc1bbd09b3a71acf467dc594fb19ada2c"}
Nov 02 15:31:00.166259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:00.166114    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn" event={"ID":"c4c9dce5-617c-4086-b13c-cd178b6bff07","Type":"ContainerStarted","Data":"1ddbe5193136157a800997a77241f9b98ae25e13d0e8de572b3fb6dbaa109231"}
Nov 02 15:31:00.395682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:00.395615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:01.177050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:01.177012    7019 generic.go:356] "Generic (PLEG): container finished" podID="c4c9dce5-617c-4086-b13c-cd178b6bff07" containerID="6714f2e9851c84253a49bcc7e46a956fc1bbd09b3a71acf467dc594fb19ada2c" exitCode=0
Nov 02 15:31:01.177268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:01.177097    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn" event={"ID":"c4c9dce5-617c-4086-b13c-cd178b6bff07","Type":"ContainerDied","Data":"6714f2e9851c84253a49bcc7e46a956fc1bbd09b3a71acf467dc594fb19ada2c"}
Nov 02 15:31:01.737548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:01.737513    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:01.958794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:01.958751    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:02.375540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.375505    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:31:02.473715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.473596    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4c9dce5-617c-4086-b13c-cd178b6bff07-kube-api-access-lmsrs\" (UniqueName: \"kubernetes.io/projected/c4c9dce5-617c-4086-b13c-cd178b6bff07-kube-api-access-lmsrs\") pod \"c4c9dce5-617c-4086-b13c-cd178b6bff07\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") "
Nov 02 15:31:02.473910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.473734    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-openshift-ca\") pod \"c4c9dce5-617c-4086-b13c-cd178b6bff07\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") "
Nov 02 15:31:02.473910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.473775    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-xks-ca\") pod \"c4c9dce5-617c-4086-b13c-cd178b6bff07\" (UID: \"c4c9dce5-617c-4086-b13c-cd178b6bff07\") "
Nov 02 15:31:02.474869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.474825    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-openshift-ca" pod "c4c9dce5-617c-4086-b13c-cd178b6bff07" (UID: "c4c9dce5-617c-4086-b13c-cd178b6bff07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:02.476610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.476582    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4c9dce5-617c-4086-b13c-cd178b6bff07-kube-api-access-lmsrs" pod "c4c9dce5-617c-4086-b13c-cd178b6bff07" (UID: "c4c9dce5-617c-4086-b13c-cd178b6bff07"). InnerVolumeSpecName "kube-api-access-lmsrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:02.476800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.476776    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-xks-ca" pod "c4c9dce5-617c-4086-b13c-cd178b6bff07" (UID: "c4c9dce5-617c-4086-b13c-cd178b6bff07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:02.576640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.576602    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmsrs\" (UniqueName: \"kubernetes.io/projected/c4c9dce5-617c-4086-b13c-cd178b6bff07-kube-api-access-lmsrs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:02.576640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.576633    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:02.576876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:02.576682    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4c9dce5-617c-4086-b13c-cd178b6bff07-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:03.197988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:03.197937    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"
Nov 02 15:31:03.197988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:03.197976    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn" event={"ID":"c4c9dce5-617c-4086-b13c-cd178b6bff07","Type":"ContainerDied","Data":"1ddbe5193136157a800997a77241f9b98ae25e13d0e8de572b3fb6dbaa109231"}
Nov 02 15:31:03.198579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:03.198030    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ddbe5193136157a800997a77241f9b98ae25e13d0e8de572b3fb6dbaa109231"
Nov 02 15:31:03.596719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:03.595749    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:05.216584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:05.216546    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:06.852017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:06.851976    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:07.228829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:07.228790    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:31:07.242037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:07.242004    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:31:08.240539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:08.240509    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:31:08.500822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:08.500720    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:10.191735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.191695    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:10.374672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.368502    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:10.792511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.792470    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"]
Nov 02 15:31:10.799589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.799552    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c4c9dce5-617c-4086-b13c-cd178b6bff07" containerName="revoke-keys"
Nov 02 15:31:10.809686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.799843    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4c9dce5-617c-4086-b13c-cd178b6bff07" containerName="revoke-keys"
Nov 02 15:31:10.810379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.810327    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4c9dce5-617c-4086-b13c-cd178b6bff07" containerName="revoke-keys"
Nov 02 15:31:10.826961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.826933    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"]
Nov 02 15:31:10.827338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.827308    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:10.883663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.883608    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:31:10.883866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.883774    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:31:10.938553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.938506    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:10.938820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.938799    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:10.939368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:10.939341    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9n2v\" (UniqueName: \"kubernetes.io/projected/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-kube-api-access-k9n2v\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.049732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.042668    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.049732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.042881    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.049732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.042937    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9n2v\" (UniqueName: \"kubernetes.io/projected/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-kube-api-access-k9n2v\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.049732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.044241    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.079100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.079043    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.096678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.093417    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9n2v\" (UniqueName: \"kubernetes.io/projected/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-kube-api-access-k9n2v\") pod \"maas-api-revoke-sub-a2c00a77-r7gfx\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.183919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.183884    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:11.981954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:11.981921    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"]
Nov 02 15:31:12.320847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:12.320718    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:12.323178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:12.323140    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx" event={"ID":"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087","Type":"ContainerStarted","Data":"35b22c15a71fc21b724956c2459d02a77faaab86680a79a769a9438cd34b9591"}
Nov 02 15:31:13.333094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:13.333003    7019 generic.go:356] "Generic (PLEG): container finished" podID="7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" containerID="ec894b606e4a1bae5501c4d6a221860d180a96850dfc6f57cc500f712b952b89" exitCode=0
Nov 02 15:31:13.333476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:13.333092    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx" event={"ID":"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087","Type":"ContainerDied","Data":"ec894b606e4a1bae5501c4d6a221860d180a96850dfc6f57cc500f712b952b89"}
Nov 02 15:31:13.964306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:13.964264    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:14.554254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.554220    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:14.583064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.583026    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-xks-ca\") pod \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") "
Nov 02 15:31:14.583260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.583127    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-kube-api-access-k9n2v\" (UniqueName: \"kubernetes.io/projected/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-kube-api-access-k9n2v\") pod \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") "
Nov 02 15:31:14.583260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.583192    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-openshift-ca\") pod \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\" (UID: \"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087\") "
Nov 02 15:31:14.584215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.583872    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-openshift-ca" pod "7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" (UID: "7f5fe9cd-f10b-474b-b341-dcc7b0e5f087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:14.584215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.584183    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:14.585472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.585435    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-xks-ca" pod "7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" (UID: "7f5fe9cd-f10b-474b-b341-dcc7b0e5f087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:14.585587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.585516    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-kube-api-access-k9n2v" pod "7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" (UID: "7f5fe9cd-f10b-474b-b341-dcc7b0e5f087"). InnerVolumeSpecName "kube-api-access-k9n2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:14.686026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.685984    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9n2v\" (UniqueName: \"kubernetes.io/projected/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-kube-api-access-k9n2v\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:14.686026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:14.686027    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:15.353873 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:15.353828    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx" event={"ID":"7f5fe9cd-f10b-474b-b341-dcc7b0e5f087","Type":"ContainerDied","Data":"35b22c15a71fc21b724956c2459d02a77faaab86680a79a769a9438cd34b9591"}
Nov 02 15:31:15.354069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:15.353901    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b22c15a71fc21b724956c2459d02a77faaab86680a79a769a9438cd34b9591"
Nov 02 15:31:15.354069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:15.353973    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"
Nov 02 15:31:15.754478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:15.754431    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:17.312007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:17.311965    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:18.909519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:18.909486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:20.584021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:20.582450    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:22.242080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.242046    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:22.366335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.366301    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"]
Nov 02 15:31:22.443033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.442986    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:22.713005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.712973    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"]
Nov 02 15:31:22.714315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.714281    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" containerName="revoke-keys"
Nov 02 15:31:22.714315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.714314    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" containerName="revoke-keys"
Nov 02 15:31:22.714587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.714571    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" containerName="revoke-keys"
Nov 02 15:31:22.718458 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:31:22.718423    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:31:22.719732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.719709    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.738206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.738176    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"]
Nov 02 15:31:22.744502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.744473    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:31:22.744760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.744741    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:31:22.848391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.848347    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.848391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.848395    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgb9f\" (UniqueName: \"kubernetes.io/projected/b8c56cdd-d4c6-4b39-946a-258e58c99259-kube-api-access-jgb9f\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.848676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.848634    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.950548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.950504    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.950846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.950826    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.950941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.950872    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgb9f\" (UniqueName: \"kubernetes.io/projected/b8c56cdd-d4c6-4b39-946a-258e58c99259-kube-api-access-jgb9f\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.951282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.951260    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.953185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.953166    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:22.975423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:22.975349    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgb9f\" (UniqueName: \"kubernetes.io/projected/b8c56cdd-d4c6-4b39-946a-258e58c99259-kube-api-access-jgb9f\") pod \"maas-api-revoke-keys-e2e-inf-a-f975ed-hp925\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:23.068500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:23.067218    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:23.392924 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:31:23.392892    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c56cdd_d4c6_4b39_946a_258e58c99259.slice/crio-e147094c07c09521ceac8cf49590535fa411bd2160815f50648fb40208fbbd5c WatchSource:0}: Error finding container e147094c07c09521ceac8cf49590535fa411bd2160815f50648fb40208fbbd5c: Status 404 returned error can't find the container with id e147094c07c09521ceac8cf49590535fa411bd2160815f50648fb40208fbbd5c
Nov 02 15:31:23.396059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:23.396027    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"]
Nov 02 15:31:23.440360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:23.440321    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925" event={"ID":"b8c56cdd-d4c6-4b39-946a-258e58c99259","Type":"ContainerStarted","Data":"e147094c07c09521ceac8cf49590535fa411bd2160815f50648fb40208fbbd5c"}
Nov 02 15:31:24.125433 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:24.125376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:24.449996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:24.449961    7019 generic.go:356] "Generic (PLEG): container finished" podID="b8c56cdd-d4c6-4b39-946a-258e58c99259" containerID="e72322e6118744c8ccc4afde82dcf0396def7e8bffb2f2f07f73b3b2c0151b6b" exitCode=0
Nov 02 15:31:24.450452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:24.450042    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925" event={"ID":"b8c56cdd-d4c6-4b39-946a-258e58c99259","Type":"ContainerDied","Data":"e72322e6118744c8ccc4afde82dcf0396def7e8bffb2f2f07f73b3b2c0151b6b"}
Nov 02 15:31:25.645346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.645310    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:25.801776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.801682    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-openshift-ca\") pod \"b8c56cdd-d4c6-4b39-946a-258e58c99259\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") "
Nov 02 15:31:25.801961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.801834    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c56cdd-d4c6-4b39-946a-258e58c99259-kube-api-access-jgb9f\" (UniqueName: \"kubernetes.io/projected/b8c56cdd-d4c6-4b39-946a-258e58c99259-kube-api-access-jgb9f\") pod \"b8c56cdd-d4c6-4b39-946a-258e58c99259\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") "
Nov 02 15:31:25.801961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.801858    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-xks-ca\") pod \"b8c56cdd-d4c6-4b39-946a-258e58c99259\" (UID: \"b8c56cdd-d4c6-4b39-946a-258e58c99259\") "
Nov 02 15:31:25.802138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.802110    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-openshift-ca" pod "b8c56cdd-d4c6-4b39-946a-258e58c99259" (UID: "b8c56cdd-d4c6-4b39-946a-258e58c99259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:25.802786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.802761    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.804036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.804013    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c56cdd-d4c6-4b39-946a-258e58c99259-kube-api-access-jgb9f" pod "b8c56cdd-d4c6-4b39-946a-258e58c99259" (UID: "b8c56cdd-d4c6-4b39-946a-258e58c99259"). InnerVolumeSpecName "kube-api-access-jgb9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:25.804036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.804026    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-xks-ca" pod "b8c56cdd-d4c6-4b39-946a-258e58c99259" (UID: "b8c56cdd-d4c6-4b39-946a-258e58c99259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:25.904826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.904791    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgb9f\" (UniqueName: \"kubernetes.io/projected/b8c56cdd-d4c6-4b39-946a-258e58c99259-kube-api-access-jgb9f\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:25.904826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:25.904821    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c56cdd-d4c6-4b39-946a-258e58c99259-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:26.042686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:26.042615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:26.469266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:26.469234    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"
Nov 02 15:31:26.469436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:26.469247    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925" event={"ID":"b8c56cdd-d4c6-4b39-946a-258e58c99259","Type":"ContainerDied","Data":"e147094c07c09521ceac8cf49590535fa411bd2160815f50648fb40208fbbd5c"}
Nov 02 15:31:26.469436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:26.469361    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e147094c07c09521ceac8cf49590535fa411bd2160815f50648fb40208fbbd5c"
Nov 02 15:31:26.709414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:26.709362    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:27.229480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:27.229446    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:27.561263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:27.561174    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:27.823274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:27.823163    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:28.010803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:28.010772    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:29.149966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:29.149929    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:30.803933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:30.803897    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:31.534663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:31.534593    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:31.704309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:31.704271    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:32.661152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:32.661113    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:33.308547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:33.308504    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:33.485768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:33.485723    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:34.020595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:34.020558    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:34.184548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:34.184505    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:34.517435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:34.517398    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:35.752622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.752590    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"]
Nov 02 15:31:35.753365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.753349    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b8c56cdd-d4c6-4b39-946a-258e58c99259" containerName="revoke-keys"
Nov 02 15:31:35.753413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.753366    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c56cdd-d4c6-4b39-946a-258e58c99259" containerName="revoke-keys"
Nov 02 15:31:35.753499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.753490    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c56cdd-d4c6-4b39-946a-258e58c99259" containerName="revoke-keys"
Nov 02 15:31:35.758860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.758837    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.779610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.779571    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"]
Nov 02 15:31:35.804983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.804911    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:31:35.805989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.805769    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:31:35.871307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.871274    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4xts\" (UniqueName: \"kubernetes.io/projected/13dd0439-d5a2-451f-ab15-cf39bca9c51f-kube-api-access-s4xts\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.871487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.871316    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.871767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.871744    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.974024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.973993    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.974240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.974145    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4xts\" (UniqueName: \"kubernetes.io/projected/13dd0439-d5a2-451f-ab15-cf39bca9c51f-kube-api-access-s4xts\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.974240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.974191    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.974939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.974912    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.976448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.976416    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:35.998906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:35.998876    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4xts\" (UniqueName: \"kubernetes.io/projected/13dd0439-d5a2-451f-ab15-cf39bca9c51f-kube-api-access-s4xts\") pod \"maas-api-revoke-sub-859260fd-545p8\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:36.077337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:36.077244    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:36.140609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:36.140568    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:36.395578 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:31:36.395347    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dd0439_d5a2_451f_ab15_cf39bca9c51f.slice/crio-ead02654c82824ff157bdab4334ab1bcc9a4618f9b3bc30364c0f2db9394d5de WatchSource:0}: Error finding container ead02654c82824ff157bdab4334ab1bcc9a4618f9b3bc30364c0f2db9394d5de: Status 404 returned error can't find the container with id ead02654c82824ff157bdab4334ab1bcc9a4618f9b3bc30364c0f2db9394d5de
Nov 02 15:31:36.398148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:36.398120    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"]
Nov 02 15:31:36.733911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:36.733819    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8" event={"ID":"13dd0439-d5a2-451f-ab15-cf39bca9c51f","Type":"ContainerStarted","Data":"ead02654c82824ff157bdab4334ab1bcc9a4618f9b3bc30364c0f2db9394d5de"}
Nov 02 15:31:37.395115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.395075    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq" podUID="816f42c0-98fd-409c-8a8c-52350a1ba4c1" containerName="main" containerID="cri-o://94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14" gracePeriod=45
Nov 02 15:31:37.717676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.717611    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:31:37.719166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.719133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:37.752727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.752690    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-home\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-home\") pod \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") "
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.762613    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tmp-dir\") pod \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") "
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.762710    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-dshm\") pod \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") "
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.762343    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-home" pod "816f42c0-98fd-409c-8a8c-52350a1ba4c1" (UID: "816f42c0-98fd-409c-8a8c-52350a1ba4c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.762747    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816f42c0-98fd-409c-8a8c-52350a1ba4c1-kube-api-access-lv25s\" (UniqueName: \"kubernetes.io/projected/816f42c0-98fd-409c-8a8c-52350a1ba4c1-kube-api-access-lv25s\") pod \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") "
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.762860    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-model-cache\") pod \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") "
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.763041    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tls-certs\") pod \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\" (UID: \"816f42c0-98fd-409c-8a8c-52350a1ba4c1\") "
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.763556    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-model-cache" pod "816f42c0-98fd-409c-8a8c-52350a1ba4c1" (UID: "816f42c0-98fd-409c-8a8c-52350a1ba4c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.763929    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.763955    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:37.769720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.764330    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tmp-dir" pod "816f42c0-98fd-409c-8a8c-52350a1ba4c1" (UID: "816f42c0-98fd-409c-8a8c-52350a1ba4c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:37.771301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.771268    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816f42c0-98fd-409c-8a8c-52350a1ba4c1-kube-api-access-lv25s" pod "816f42c0-98fd-409c-8a8c-52350a1ba4c1" (UID: "816f42c0-98fd-409c-8a8c-52350a1ba4c1"). InnerVolumeSpecName "kube-api-access-lv25s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:37.771448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.771346    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-dshm" pod "816f42c0-98fd-409c-8a8c-52350a1ba4c1" (UID: "816f42c0-98fd-409c-8a8c-52350a1ba4c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:31:37.771448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.771400    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tls-certs" pod "816f42c0-98fd-409c-8a8c-52350a1ba4c1" (UID: "816f42c0-98fd-409c-8a8c-52350a1ba4c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:37.819014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.811217    7019 generic.go:356] "Generic (PLEG): container finished" podID="13dd0439-d5a2-451f-ab15-cf39bca9c51f" containerID="8092a173dff4a481b4f2241570407a5b5c2116e14ee693192d3fc48b9b997060" exitCode=0
Nov 02 15:31:37.819014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.811483    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8" event={"ID":"13dd0439-d5a2-451f-ab15-cf39bca9c51f","Type":"ContainerDied","Data":"8092a173dff4a481b4f2241570407a5b5c2116e14ee693192d3fc48b9b997060"}
Nov 02 15:31:37.819014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.813407    7019 generic.go:356] "Generic (PLEG): container finished" podID="816f42c0-98fd-409c-8a8c-52350a1ba4c1" containerID="94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14" exitCode=0
Nov 02 15:31:37.819014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.813486    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq" event={"ID":"816f42c0-98fd-409c-8a8c-52350a1ba4c1","Type":"ContainerDied","Data":"94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14"}
Nov 02 15:31:37.819014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.813511    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq" event={"ID":"816f42c0-98fd-409c-8a8c-52350a1ba4c1","Type":"ContainerDied","Data":"42405957718f54c14188b56f79eed585c8b1f9b19a239c3cfec73ed027a90810"}
Nov 02 15:31:37.819014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.813527    7019 scope.go:122] "RemoveContainer" containerID="94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14"
Nov 02 15:31:37.819014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.813659    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"
Nov 02 15:31:37.836982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.836954    7019 scope.go:122] "RemoveContainer" containerID="94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14"
Nov 02 15:31:37.837432 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:31:37.837402    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14\": container with ID starting with 94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14 not found: ID does not exist" containerID="94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14"
Nov 02 15:31:37.837570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.837444    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14"} err="failed to get container status \"94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14\": rpc error: code = NotFound desc = could not find container \"94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14\": container with ID starting with 94ee785128cd9dd2e9abcd1c01016ca804f5e96b22266344513b4a6e0c94ad14 not found: ID does not exist"
Nov 02 15:31:37.865534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.865493    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:37.865534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.865528    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/816f42c0-98fd-409c-8a8c-52350a1ba4c1-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:37.866215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.865541    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv25s\" (UniqueName: \"kubernetes.io/projected/816f42c0-98fd-409c-8a8c-52350a1ba4c1-kube-api-access-lv25s\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:37.866215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.865556    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/816f42c0-98fd-409c-8a8c-52350a1ba4c1-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:37.896416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.896353    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"]
Nov 02 15:31:37.902579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:37.902546    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2abeaa/test-model-2abeaa-kserve-754b74f5f-4dqsq"]
Nov 02 15:31:39.008345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.008313    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:39.087955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.087852    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-openshift-ca\") pod \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") "
Nov 02 15:31:39.088138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.088015    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-xks-ca\") pod \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") "
Nov 02 15:31:39.088298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.088275    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13dd0439-d5a2-451f-ab15-cf39bca9c51f-kube-api-access-s4xts\" (UniqueName: \"kubernetes.io/projected/13dd0439-d5a2-451f-ab15-cf39bca9c51f-kube-api-access-s4xts\") pod \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\" (UID: \"13dd0439-d5a2-451f-ab15-cf39bca9c51f\") "
Nov 02 15:31:39.088380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.088316    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-openshift-ca" pod "13dd0439-d5a2-451f-ab15-cf39bca9c51f" (UID: "13dd0439-d5a2-451f-ab15-cf39bca9c51f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:39.089203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.089166    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:39.090972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.090935    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-xks-ca" pod "13dd0439-d5a2-451f-ab15-cf39bca9c51f" (UID: "13dd0439-d5a2-451f-ab15-cf39bca9c51f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:39.091249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.091216    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dd0439-d5a2-451f-ab15-cf39bca9c51f-kube-api-access-s4xts" pod "13dd0439-d5a2-451f-ab15-cf39bca9c51f" (UID: "13dd0439-d5a2-451f-ab15-cf39bca9c51f"). InnerVolumeSpecName "kube-api-access-s4xts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:39.190828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.190790    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13dd0439-d5a2-451f-ab15-cf39bca9c51f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:39.190828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.190822    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4xts\" (UniqueName: \"kubernetes.io/projected/13dd0439-d5a2-451f-ab15-cf39bca9c51f-kube-api-access-s4xts\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:39.299319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.299282    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:39.833874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.833845    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"
Nov 02 15:31:39.834062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.833886    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8" event={"ID":"13dd0439-d5a2-451f-ab15-cf39bca9c51f","Type":"ContainerDied","Data":"ead02654c82824ff157bdab4334ab1bcc9a4618f9b3bc30364c0f2db9394d5de"}
Nov 02 15:31:39.834062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.833943    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ead02654c82824ff157bdab4334ab1bcc9a4618f9b3bc30364c0f2db9394d5de"
Nov 02 15:31:39.865082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:39.865047    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816f42c0-98fd-409c-8a8c-52350a1ba4c1" path="/var/lib/kubelet/pods/816f42c0-98fd-409c-8a8c-52350a1ba4c1/volumes"
Nov 02 15:31:40.932676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:40.932418    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:42.459308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:42.459265    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:44.106433 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:44.106395    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:45.695426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:45.695361    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:47.113662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:47.113617    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:31:47.322365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:47.322322    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:48.418203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:48.418167    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:48.484673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:48.462030    7019 scope.go:122] "RemoveContainer" containerID="33735ea992cd905b3c3c58ffef036121cf0cbfe92ba12ab9371df29701ccf4fc"
Nov 02 15:31:48.540668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:48.536882    7019 scope.go:122] "RemoveContainer" containerID="183574c56e4a37e06c7b2e2e3c805b3463935b30091d8cb4fcbd8c221064d996"
Nov 02 15:31:48.603929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:48.603903    7019 scope.go:122] "RemoveContainer" containerID="4b36bba05a05151a42389480b533b4b6b2b7405c16a74c092590e98f4b446f29"
Nov 02 15:31:49.289898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:49.289864    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:51.191420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:51.191377    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:52.713455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:52.713420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:52.715384 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:31:52.715352    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:31:53.155084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:53.155042    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:54.401972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.401944    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:54.562671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.562201    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:54.976469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.976433    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"]
Nov 02 15:31:54.977279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.977258    7019 cpu_manager.go:414] "containerMap: removing container" podUID="816f42c0-98fd-409c-8a8c-52350a1ba4c1" containerName="main"
Nov 02 15:31:54.977339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.977280    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816f42c0-98fd-409c-8a8c-52350a1ba4c1" containerName="main"
Nov 02 15:31:54.977339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.977328    7019 cpu_manager.go:414] "containerMap: removing container" podUID="13dd0439-d5a2-451f-ab15-cf39bca9c51f" containerName="revoke-keys"
Nov 02 15:31:54.977339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.977334    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13dd0439-d5a2-451f-ab15-cf39bca9c51f" containerName="revoke-keys"
Nov 02 15:31:54.977460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.977451    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="816f42c0-98fd-409c-8a8c-52350a1ba4c1" containerName="main"
Nov 02 15:31:54.977497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.977469    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="13dd0439-d5a2-451f-ab15-cf39bca9c51f" containerName="revoke-keys"
Nov 02 15:31:54.983445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:54.983420    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.002824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.002409    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"]
Nov 02 15:31:55.008733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.008634    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:31:55.008886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.008862    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:31:55.044222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.044185    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xz9\" (UniqueName: \"kubernetes.io/projected/aef69a49-ec78-4a4e-8063-aff678a9ccb1-kube-api-access-78xz9\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.044409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.044308    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.044483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.044435    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.145586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.145544    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xz9\" (UniqueName: \"kubernetes.io/projected/aef69a49-ec78-4a4e-8063-aff678a9ccb1-kube-api-access-78xz9\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.145833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.145617    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.145833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.145738    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.146449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.146422    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.148160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.148129    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.171033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.170999    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xz9\" (UniqueName: \"kubernetes.io/projected/aef69a49-ec78-4a4e-8063-aff678a9ccb1-kube-api-access-78xz9\") pod \"maas-api-revoke-sub-01de903b-zv248\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.304790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.304697    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:55.613932 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:31:55.613886    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef69a49_ec78_4a4e_8063_aff678a9ccb1.slice/crio-ad5bd140ad9fe808164668a99739f6c366865a7b94aae9a8087603c8a3a68988 WatchSource:0}: Error finding container ad5bd140ad9fe808164668a99739f6c366865a7b94aae9a8087603c8a3a68988: Status 404 returned error can't find the container with id ad5bd140ad9fe808164668a99739f6c366865a7b94aae9a8087603c8a3a68988
Nov 02 15:31:55.616060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:55.616031    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"]
Nov 02 15:31:56.201714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:56.197769    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:56.228400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:56.227475    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248" event={"ID":"aef69a49-ec78-4a4e-8063-aff678a9ccb1","Type":"ContainerStarted","Data":"e30226dfe8d1f639c24dda15bb512d8d7410b37bf663b674e18d29ec1a4d13ce"}
Nov 02 15:31:56.228400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:56.227536    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248" event={"ID":"aef69a49-ec78-4a4e-8063-aff678a9ccb1","Type":"ContainerStarted","Data":"ad5bd140ad9fe808164668a99739f6c366865a7b94aae9a8087603c8a3a68988"}
Nov 02 15:31:56.270703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:56.270626    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248" podStartSLOduration=2.270612012 podStartE2EDuration="2.270612012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:56.269239417 +0000 UTC m=+2110.523193042" watchObservedRunningTime="2026-11-02 15:31:56.270612012 +0000 UTC m=+2110.524565660"
Nov 02 15:31:57.239182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:57.239149    7019 generic.go:356] "Generic (PLEG): container finished" podID="aef69a49-ec78-4a4e-8063-aff678a9ccb1" containerID="e30226dfe8d1f639c24dda15bb512d8d7410b37bf663b674e18d29ec1a4d13ce" exitCode=0
Nov 02 15:31:57.239660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:57.239302    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248" event={"ID":"aef69a49-ec78-4a4e-8063-aff678a9ccb1","Type":"ContainerDied","Data":"e30226dfe8d1f639c24dda15bb512d8d7410b37bf663b674e18d29ec1a4d13ce"}
Nov 02 15:31:57.789092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:57.789057    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:58.446619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.446591    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:58.501850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.501751    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-xks-ca\") pod \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") "
Nov 02 15:31:58.502026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.501915    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-openshift-ca\") pod \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") "
Nov 02 15:31:58.502026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.501987    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef69a49-ec78-4a4e-8063-aff678a9ccb1-kube-api-access-78xz9\" (UniqueName: \"kubernetes.io/projected/aef69a49-ec78-4a4e-8063-aff678a9ccb1-kube-api-access-78xz9\") pod \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\" (UID: \"aef69a49-ec78-4a4e-8063-aff678a9ccb1\") "
Nov 02 15:31:58.502463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.502434    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-openshift-ca" pod "aef69a49-ec78-4a4e-8063-aff678a9ccb1" (UID: "aef69a49-ec78-4a4e-8063-aff678a9ccb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:31:58.502821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.502796    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:58.504509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.504481    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-xks-ca" pod "aef69a49-ec78-4a4e-8063-aff678a9ccb1" (UID: "aef69a49-ec78-4a4e-8063-aff678a9ccb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:31:58.509832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.509793    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef69a49-ec78-4a4e-8063-aff678a9ccb1-kube-api-access-78xz9" pod "aef69a49-ec78-4a4e-8063-aff678a9ccb1" (UID: "aef69a49-ec78-4a4e-8063-aff678a9ccb1"). InnerVolumeSpecName "kube-api-access-78xz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:31:58.604597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.604562    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aef69a49-ec78-4a4e-8063-aff678a9ccb1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:58.604597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:58.604593    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xz9\" (UniqueName: \"kubernetes.io/projected/aef69a49-ec78-4a4e-8063-aff678a9ccb1-kube-api-access-78xz9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:31:59.259617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:59.259584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:31:59.261721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:59.261692    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248" event={"ID":"aef69a49-ec78-4a4e-8063-aff678a9ccb1","Type":"ContainerDied","Data":"ad5bd140ad9fe808164668a99739f6c366865a7b94aae9a8087603c8a3a68988"}
Nov 02 15:31:59.261880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:59.261755    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5bd140ad9fe808164668a99739f6c366865a7b94aae9a8087603c8a3a68988"
Nov 02 15:31:59.261880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:59.261704    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"
Nov 02 15:31:59.883425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:31:59.865136    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:00.034555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:00.034516    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"]
Nov 02 15:32:00.037638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:00.037611    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f975ed-hp925"]
Nov 02 15:32:00.059404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:00.059367    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:00.409078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:00.409030    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:00.582487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:00.582449    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:01.864362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:01.864328    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c56cdd-d4c6-4b39-946a-258e58c99259" path="/var/lib/kubelet/pods/b8c56cdd-d4c6-4b39-946a-258e58c99259/volumes"
Nov 02 15:32:01.866405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:01.866378    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:02.149303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:02.149188    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:03.803881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:03.803841    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:05.314809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:05.314756    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:06.992413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:06.992371    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:07.172444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:07.172402    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:07.331898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:07.331807    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:07.458939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:07.458889    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"]
Nov 02 15:32:07.459195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:07.459168    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy" containerID="cri-o://b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0" gracePeriod=30
Nov 02 15:32:07.511149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:07.511116    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:08.081913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.081872    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"]
Nov 02 15:32:08.083029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.083002    7019 cpu_manager.go:414] "containerMap: removing container" podUID="aef69a49-ec78-4a4e-8063-aff678a9ccb1" containerName="revoke-keys"
Nov 02 15:32:08.083029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.083028    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef69a49-ec78-4a4e-8063-aff678a9ccb1" containerName="revoke-keys"
Nov 02 15:32:08.083318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.083305    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef69a49-ec78-4a4e-8063-aff678a9ccb1" containerName="revoke-keys"
Nov 02 15:32:08.089719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.089694    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.105341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.105314    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"]
Nov 02 15:32:08.146260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.146222    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb77z\" (UniqueName: \"kubernetes.io/projected/8de6019c-106f-4015-9876-b8b4ba2de54b-kube-api-access-tb77z\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.146447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.146287    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.146497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.146470    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.154176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.154145    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:32:08.154315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.154200    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:32:08.247725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.247689    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.247903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.247818    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb77z\" (UniqueName: \"kubernetes.io/projected/8de6019c-106f-4015-9876-b8b4ba2de54b-kube-api-access-tb77z\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.247903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.247866    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.248436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.248415    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.250346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.250329    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.272776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.272749    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb77z\" (UniqueName: \"kubernetes.io/projected/8de6019c-106f-4015-9876-b8b4ba2de54b-kube-api-access-tb77z\") pod \"maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.408667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.408612    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:08.768142 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:32:08.768099    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de6019c_106f_4015_9876_b8b4ba2de54b.slice/crio-f4b64fba75930435a770866b9fb3ce2d23d581ed041ae00afa5ed54477c558de WatchSource:0}: Error finding container f4b64fba75930435a770866b9fb3ce2d23d581ed041ae00afa5ed54477c558de: Status 404 returned error can't find the container with id f4b64fba75930435a770866b9fb3ce2d23d581ed041ae00afa5ed54477c558de
Nov 02 15:32:08.769811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.769784    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"]
Nov 02 15:32:08.870992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:08.870958    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:09.033589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:09.033487    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"]
Nov 02 15:32:09.037464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:09.037435    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b034def-7wm9f"]
Nov 02 15:32:09.420551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:09.420504    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt" event={"ID":"8de6019c-106f-4015-9876-b8b4ba2de54b","Type":"ContainerStarted","Data":"945ef0c65e242f8c62c45c397ce42bf35da119d7ce0256358b14dc623b12f585"}
Nov 02 15:32:09.421158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:09.420572    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt" event={"ID":"8de6019c-106f-4015-9876-b8b4ba2de54b","Type":"ContainerStarted","Data":"f4b64fba75930435a770866b9fb3ce2d23d581ed041ae00afa5ed54477c558de"}
Nov 02 15:32:09.463728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:09.463633    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt" podStartSLOduration=1.463619143 podStartE2EDuration="1.463619143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:09.462464679 +0000 UTC m=+2123.716418303" watchObservedRunningTime="2026-11-02 15:32:09.463619143 +0000 UTC m=+2123.717572783"
Nov 02 15:32:09.880229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:09.880185    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d765c83-2f44-450c-8347-5ada14347576" path="/var/lib/kubelet/pods/1d765c83-2f44-450c-8347-5ada14347576/volumes"
Nov 02 15:32:10.430282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:10.430250    7019 generic.go:356] "Generic (PLEG): container finished" podID="8de6019c-106f-4015-9876-b8b4ba2de54b" containerID="945ef0c65e242f8c62c45c397ce42bf35da119d7ce0256358b14dc623b12f585" exitCode=0
Nov 02 15:32:10.430687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:10.430329    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt" event={"ID":"8de6019c-106f-4015-9876-b8b4ba2de54b","Type":"ContainerDied","Data":"945ef0c65e242f8c62c45c397ce42bf35da119d7ce0256358b14dc623b12f585"}
Nov 02 15:32:10.662279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:10.662232    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:11.246525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.246492    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:11.492144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.491803    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"]
Nov 02 15:32:11.498023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.497955    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.516563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.516533    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"]
Nov 02 15:32:11.531438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.529812    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.532413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.532384    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.532766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.532734    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th5lx\" (UniqueName: \"kubernetes.io/projected/e02c30ad-e23d-489d-bdac-a0af77b9566c-kube-api-access-th5lx\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.642675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.639037    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.642675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.639364    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.642675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.639439    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th5lx\" (UniqueName: \"kubernetes.io/projected/e02c30ad-e23d-489d-bdac-a0af77b9566c-kube-api-access-th5lx\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.643010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.642938    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.651589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.651549    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.666927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.665714    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th5lx\" (UniqueName: \"kubernetes.io/projected/e02c30ad-e23d-489d-bdac-a0af77b9566c-kube-api-access-th5lx\") pod \"maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.724826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.724788    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:11.820439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.820334    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:11.844670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.844525    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-xks-ca\") pod \"8de6019c-106f-4015-9876-b8b4ba2de54b\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") "
Nov 02 15:32:11.844809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.844680    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-openshift-ca\") pod \"8de6019c-106f-4015-9876-b8b4ba2de54b\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") "
Nov 02 15:32:11.845159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.845130    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-openshift-ca" pod "8de6019c-106f-4015-9876-b8b4ba2de54b" (UID: "8de6019c-106f-4015-9876-b8b4ba2de54b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:11.845380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.845339    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8de6019c-106f-4015-9876-b8b4ba2de54b-kube-api-access-tb77z\" (UniqueName: \"kubernetes.io/projected/8de6019c-106f-4015-9876-b8b4ba2de54b-kube-api-access-tb77z\") pod \"8de6019c-106f-4015-9876-b8b4ba2de54b\" (UID: \"8de6019c-106f-4015-9876-b8b4ba2de54b\") "
Nov 02 15:32:11.846575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.846548    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:11.847102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.847073    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-xks-ca" pod "8de6019c-106f-4015-9876-b8b4ba2de54b" (UID: "8de6019c-106f-4015-9876-b8b4ba2de54b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:11.853040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.853016    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de6019c-106f-4015-9876-b8b4ba2de54b-kube-api-access-tb77z" pod "8de6019c-106f-4015-9876-b8b4ba2de54b" (UID: "8de6019c-106f-4015-9876-b8b4ba2de54b"). InnerVolumeSpecName "kube-api-access-tb77z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:11.949194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.949154    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8de6019c-106f-4015-9876-b8b4ba2de54b-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:11.949194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:11.949185    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb77z\" (UniqueName: \"kubernetes.io/projected/8de6019c-106f-4015-9876-b8b4ba2de54b-kube-api-access-tb77z\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.151810 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:32:12.151781    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02c30ad_e23d_489d_bdac_a0af77b9566c.slice/crio-82c0524ff5c26441f243a94f90080e5bb30857a81843ac1d4f799d687159a0e9 WatchSource:0}: Error finding container 82c0524ff5c26441f243a94f90080e5bb30857a81843ac1d4f799d687159a0e9: Status 404 returned error can't find the container with id 82c0524ff5c26441f243a94f90080e5bb30857a81843ac1d4f799d687159a0e9
Nov 02 15:32:12.152965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.152939    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"]
Nov 02 15:32:12.458849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.458747    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z" event={"ID":"e02c30ad-e23d-489d-bdac-a0af77b9566c","Type":"ContainerStarted","Data":"82c0524ff5c26441f243a94f90080e5bb30857a81843ac1d4f799d687159a0e9"}
Nov 02 15:32:12.460437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.460411    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt" event={"ID":"8de6019c-106f-4015-9876-b8b4ba2de54b","Type":"ContainerDied","Data":"f4b64fba75930435a770866b9fb3ce2d23d581ed041ae00afa5ed54477c558de"}
Nov 02 15:32:12.460556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.460453    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b64fba75930435a770866b9fb3ce2d23d581ed041ae00afa5ed54477c558de"
Nov 02 15:32:12.460556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.460422    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"
Nov 02 15:32:12.513673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.512404    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:12.798758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.798726    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:32:12.865046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865006    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istiod-ca-cert\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865151    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-envoy\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865269    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-certs\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865318    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-socket\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865487    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-credential-socket\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865586    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-data\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865639    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-kube-api-access-2zfb2\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-kube-api-access-2zfb2\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865685    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-podinfo\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865759    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-token\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-token\") pod \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\" (UID: \"377eabb1-c526-4a6b-9d85-05d6004a0cf9\") "
Nov 02 15:32:12.865835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865807    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-socket" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:12.865956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.865733    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istiod-ca-cert" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:12.866309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.866152    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-data" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:12.866309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.866195    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-certs" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:12.866564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.866537    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-credential-socket" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:12.866704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.866557    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.866853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.866837    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.866960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.866948    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.867052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.867040    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.868900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.868847    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-podinfo" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:12.868900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.868869    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-kube-api-access-2zfb2" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "kube-api-access-2zfb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:12.869958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.869929    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-token" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:12.870252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.870213    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-envoy" pod "377eabb1-c526-4a6b-9d85-05d6004a0cf9" (UID: "377eabb1-c526-4a6b-9d85-05d6004a0cf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:12.968738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.968693    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.968738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.968731    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zfb2\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-kube-api-access-2zfb2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.968738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.968747    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.969091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.968764    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:12.969091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:12.968778    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377eabb1-c526-4a6b-9d85-05d6004a0cf9-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:13.471845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.471794    7019 generic.go:356] "Generic (PLEG): container finished" podID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerID="b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0" exitCode=0
Nov 02 15:32:13.472040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.471853    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"
Nov 02 15:32:13.472040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.471861    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" event={"ID":"377eabb1-c526-4a6b-9d85-05d6004a0cf9","Type":"ContainerDied","Data":"b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0"}
Nov 02 15:32:13.472040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.471928    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5" event={"ID":"377eabb1-c526-4a6b-9d85-05d6004a0cf9","Type":"ContainerDied","Data":"73f040a5953e081abd4b90a3af55ee51a36e3510bcdbed1c5dc211cf22ba2761"}
Nov 02 15:32:13.472040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.471960    7019 scope.go:122] "RemoveContainer" containerID="b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0"
Nov 02 15:32:13.473801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.473778    7019 generic.go:356] "Generic (PLEG): container finished" podID="e02c30ad-e23d-489d-bdac-a0af77b9566c" containerID="bc72235027db4a68dfa9fdce9703a0722c7c9470d43bb5a767b7f068215425d6" exitCode=0
Nov 02 15:32:13.473893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.473855    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z" event={"ID":"e02c30ad-e23d-489d-bdac-a0af77b9566c","Type":"ContainerDied","Data":"bc72235027db4a68dfa9fdce9703a0722c7c9470d43bb5a767b7f068215425d6"}
Nov 02 15:32:13.485219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.485201    7019 scope.go:122] "RemoveContainer" containerID="775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf"
Nov 02 15:32:13.495542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.495523    7019 scope.go:122] "RemoveContainer" containerID="b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0"
Nov 02 15:32:13.495828 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:32:13.495807    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0\": container with ID starting with b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0 not found: ID does not exist" containerID="b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0"
Nov 02 15:32:13.495890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.495838    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0"} err="failed to get container status \"b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0\": rpc error: code = NotFound desc = could not find container \"b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0\": container with ID starting with b58bf4eb7c74da942cf0b31624f2a040889614bd540010b7b0050fff48999be0 not found: ID does not exist"
Nov 02 15:32:13.495890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.495856    7019 scope.go:122] "RemoveContainer" containerID="775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf"
Nov 02 15:32:13.496107 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:32:13.496089    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf\": container with ID starting with 775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf not found: ID does not exist" containerID="775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf"
Nov 02 15:32:13.496152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.496110    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf"} err="failed to get container status \"775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf\": rpc error: code = NotFound desc = could not find container \"775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf\": container with ID starting with 775aba3ccb595f079d3ba60a3ac6c8cb1fde1693e6211238eaffc621fe5dfebf not found: ID does not exist"
Nov 02 15:32:13.583588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.583496    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"]
Nov 02 15:32:13.587403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.587376    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f975ed-openshift-default-76b6467db7-hcmm5"]
Nov 02 15:32:13.864784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:13.864750    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" path="/var/lib/kubelet/pods/377eabb1-c526-4a6b-9d85-05d6004a0cf9/volumes"
Nov 02 15:32:14.105280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.105238    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"]
Nov 02 15:32:14.106017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.105998    7019 cpu_manager.go:414] "containerMap: removing container" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy"
Nov 02 15:32:14.106017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106018    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy"
Nov 02 15:32:14.106113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106056    7019 cpu_manager.go:414] "containerMap: removing container" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy"
Nov 02 15:32:14.106113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106063    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy"
Nov 02 15:32:14.106113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106071    7019 cpu_manager.go:414] "containerMap: removing container" podUID="8de6019c-106f-4015-9876-b8b4ba2de54b" containerName="revoke-keys"
Nov 02 15:32:14.106113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106077    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8de6019c-106f-4015-9876-b8b4ba2de54b" containerName="revoke-keys"
Nov 02 15:32:14.106247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106183    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy"
Nov 02 15:32:14.106247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106199    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="377eabb1-c526-4a6b-9d85-05d6004a0cf9" containerName="istio-proxy"
Nov 02 15:32:14.106247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.106206    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8de6019c-106f-4015-9876-b8b4ba2de54b" containerName="revoke-keys"
Nov 02 15:32:14.111400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.111371    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.130294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.130265    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-28f621-openshift-default-dockercfg-qv46v\""
Nov 02 15:32:14.135256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.135228    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"]
Nov 02 15:32:14.187809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.187771    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-certs\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.187819    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-credential-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.187885    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b46866-6bf6-4935-ae33-23d0c1579263-istiod-ca-cert\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.187922    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.187999    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-podinfo\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.188183    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-data\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.188353    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmddn\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-kube-api-access-fmddn\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.188377    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-envoy\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.188498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.188426    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-token\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.289581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289542    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmddn\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-kube-api-access-fmddn\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.289581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289583    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-envoy\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.289870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289621    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-token\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.289870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289687    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-certs\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.289870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289723    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-credential-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.289870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289806    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b46866-6bf6-4935-ae33-23d0c1579263-istiod-ca-cert\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.289870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289864    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.290148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.289965    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-podinfo\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.290148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.290008    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-data\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.290260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.290189    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-certs\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.290260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.290211    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-credential-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.290394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.290371    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-socket\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.290564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.290544    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-data\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.290776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.290748    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b46866-6bf6-4935-ae33-23d0c1579263-istiod-ca-cert\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.292360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.292335    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-envoy\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.292544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.292515    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:14.292899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.292879    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-podinfo\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.322706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.317667    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-token\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.322706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.319469    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmddn\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-kube-api-access-fmddn\") pod \"e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") " pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.426079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.426002    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:14.722442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.722412    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:14.756573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.756539    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"]
Nov 02 15:32:14.772721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.772574    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:32:14.772721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.772691    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:32:14.772937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.772733    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:32:14.805615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.805571    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-openshift-ca\") pod \"e02c30ad-e23d-489d-bdac-a0af77b9566c\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") "
Nov 02 15:32:14.806016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.805992    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-xks-ca\") pod \"e02c30ad-e23d-489d-bdac-a0af77b9566c\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") "
Nov 02 15:32:14.806251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.806088    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-openshift-ca" pod "e02c30ad-e23d-489d-bdac-a0af77b9566c" (UID: "e02c30ad-e23d-489d-bdac-a0af77b9566c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:14.806570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.806546    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02c30ad-e23d-489d-bdac-a0af77b9566c-kube-api-access-th5lx\" (UniqueName: \"kubernetes.io/projected/e02c30ad-e23d-489d-bdac-a0af77b9566c-kube-api-access-th5lx\") pod \"e02c30ad-e23d-489d-bdac-a0af77b9566c\" (UID: \"e02c30ad-e23d-489d-bdac-a0af77b9566c\") "
Nov 02 15:32:14.807563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.807538    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.811312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.811279    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02c30ad-e23d-489d-bdac-a0af77b9566c-kube-api-access-th5lx" pod "e02c30ad-e23d-489d-bdac-a0af77b9566c" (UID: "e02c30ad-e23d-489d-bdac-a0af77b9566c"). InnerVolumeSpecName "kube-api-access-th5lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:14.811572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.811550    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-xks-ca" pod "e02c30ad-e23d-489d-bdac-a0af77b9566c" (UID: "e02c30ad-e23d-489d-bdac-a0af77b9566c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:14.912003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.911962    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e02c30ad-e23d-489d-bdac-a0af77b9566c-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:14.912003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:14.912006    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th5lx\" (UniqueName: \"kubernetes.io/projected/e02c30ad-e23d-489d-bdac-a0af77b9566c-kube-api-access-th5lx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:15.505728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:15.505684    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" event={"ID":"b4b46866-6bf6-4935-ae33-23d0c1579263","Type":"ContainerStarted","Data":"65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f"}
Nov 02 15:32:15.505948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:15.505747    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" event={"ID":"b4b46866-6bf6-4935-ae33-23d0c1579263","Type":"ContainerStarted","Data":"e64687436a27cb1753b9aea06bb6fa83d1a2b27d35dd08d5738186fec8547ab1"}
Nov 02 15:32:15.507415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:15.507389    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"
Nov 02 15:32:15.507561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:15.507391    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z" event={"ID":"e02c30ad-e23d-489d-bdac-a0af77b9566c","Type":"ContainerDied","Data":"82c0524ff5c26441f243a94f90080e5bb30857a81843ac1d4f799d687159a0e9"}
Nov 02 15:32:15.507561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:15.507535    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c0524ff5c26441f243a94f90080e5bb30857a81843ac1d4f799d687159a0e9"
Nov 02 15:32:15.556426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:15.556364    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" podStartSLOduration=1.55634914 podStartE2EDuration="1.55634914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:15.552564051 +0000 UTC m=+2129.806517683" watchObservedRunningTime="2026-11-02 15:32:15.55634914 +0000 UTC m=+2129.810302764"
Nov 02 15:32:16.062105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:16.062068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:16.427063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:16.427018    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:16.428846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:16.428817    7019 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 15:32:16.428969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:16.428871    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 15:32:17.426717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:17.426629    7019 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 15:32:17.427146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:17.426750    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 15:32:17.939027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:17.938990    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:18.427342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:18.427297    7019 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 15:32:18.427829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:18.427373    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 15:32:18.754662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:18.754611    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:18.905365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:18.905027    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:19.435670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:19.435077    7019 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 15:32:19.435670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:19.435142    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 15:32:19.652974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:19.652740    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:19.807728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:19.801189    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:19.947493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:19.947439    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:20.432784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:20.427144    7019 patch_prober.go:29] interesting pod/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Nov 02 15:32:20.432784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:20.427226    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Nov 02 15:32:21.037555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:21.037519    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"]
Nov 02 15:32:21.050195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:21.050159    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d161ec86-l2nz2"]
Nov 02 15:32:21.388093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:21.388057    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:21.661377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:21.661284    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:21.661987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:21.661961    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:21.662986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:21.662964    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:32:21.865384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:21.865350    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1d3c7b-e5ae-4385-b38b-32efa518118c" path="/var/lib/kubelet/pods/8c1d3c7b-e5ae-4385-b38b-32efa518118c/volumes"
Nov 02 15:32:22.131897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:22.131867    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:22.280843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:22.280808    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:22.829195 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:32:22.829156    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:32:23.091824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:23.091736    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:24.734807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:24.734765    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:26.295563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:26.295517    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:27.913509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:27.913473    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:29.602067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:29.602016    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:31.180886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:31.180830    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:32.524063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:32.524019    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:33.123080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:33.123043    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:33.185826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:33.185782    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"]
Nov 02 15:32:33.193521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:33.193492    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a03e84-4jm6z"]
Nov 02 15:32:33.276719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:33.268811    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:33.685503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:33.685472    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:33.863769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:33.863735    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02c30ad-e23d-489d-bdac-a0af77b9566c" path="/var/lib/kubelet/pods/e02c30ad-e23d-489d-bdac-a0af77b9566c/volumes"
Nov 02 15:32:34.998800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:34.996377    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:35.782337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:35.782295    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:36.627659 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:36.627588    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:38.209211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:38.209160    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:39.218978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:39.218933    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:39.888064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:39.888017    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:40.043082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.043047    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:40.349514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.349424    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"]
Nov 02 15:32:40.350030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.349682    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy" containerID="cri-o://c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096" gracePeriod=30
Nov 02 15:32:40.565246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.565164    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:40.901958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.901915    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"]
Nov 02 15:32:40.903200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.903175    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e02c30ad-e23d-489d-bdac-a0af77b9566c" containerName="revoke-keys"
Nov 02 15:32:40.903278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.903203    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02c30ad-e23d-489d-bdac-a0af77b9566c" containerName="revoke-keys"
Nov 02 15:32:40.903390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.903378    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02c30ad-e23d-489d-bdac-a0af77b9566c" containerName="revoke-keys"
Nov 02 15:32:40.908006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.907981    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:40.924511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.924459    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"]
Nov 02 15:32:40.924705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.924524    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:32:40.924705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:40.924594    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:32:41.006253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.006215    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.006481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.006452    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.006730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.006701    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzwrg\" (UniqueName: \"kubernetes.io/projected/6585fca7-1520-47ab-ad9b-fb683ef63e67-kube-api-access-fzwrg\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.108127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.108078    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzwrg\" (UniqueName: \"kubernetes.io/projected/6585fca7-1520-47ab-ad9b-fb683ef63e67-kube-api-access-fzwrg\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.108333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.108281    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.108476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.108448    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.109324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.109297    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.111072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.111049    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.136581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.136553    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzwrg\" (UniqueName: \"kubernetes.io/projected/6585fca7-1520-47ab-ad9b-fb683ef63e67-kube-api-access-fzwrg\") pod \"maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.227073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.226981    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:41.544421 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:32:41.544390    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6585fca7_1520_47ab_ad9b_fb683ef63e67.slice/crio-b975d14687d6c9084015cb32e92b4fe9e85f06e7c93fc377bb5eee9d4a742845 WatchSource:0}: Error finding container b975d14687d6c9084015cb32e92b4fe9e85f06e7c93fc377bb5eee9d4a742845: Status 404 returned error can't find the container with id b975d14687d6c9084015cb32e92b4fe9e85f06e7c93fc377bb5eee9d4a742845
Nov 02 15:32:41.545720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.545695    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"]
Nov 02 15:32:41.875583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.875541    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj" event={"ID":"6585fca7-1520-47ab-ad9b-fb683ef63e67","Type":"ContainerStarted","Data":"b975d14687d6c9084015cb32e92b4fe9e85f06e7c93fc377bb5eee9d4a742845"}
Nov 02 15:32:41.933134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:41.933095    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:42.887474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:42.887433    7019 generic.go:356] "Generic (PLEG): container finished" podID="6585fca7-1520-47ab-ad9b-fb683ef63e67" containerID="a6e402a7da5154962965e2b1ef723015717bfec5590810dd7b939af43508ac01" exitCode=0
Nov 02 15:32:42.887906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:42.887521    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj" event={"ID":"6585fca7-1520-47ab-ad9b-fb683ef63e67","Type":"ContainerDied","Data":"a6e402a7da5154962965e2b1ef723015717bfec5590810dd7b939af43508ac01"}
Nov 02 15:32:43.607176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:43.606951    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:44.116076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.116043    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:44.174365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.174279    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-xks-ca\") pod \"6585fca7-1520-47ab-ad9b-fb683ef63e67\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") "
Nov 02 15:32:44.175156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.175130    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6585fca7-1520-47ab-ad9b-fb683ef63e67-kube-api-access-fzwrg\" (UniqueName: \"kubernetes.io/projected/6585fca7-1520-47ab-ad9b-fb683ef63e67-kube-api-access-fzwrg\") pod \"6585fca7-1520-47ab-ad9b-fb683ef63e67\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") "
Nov 02 15:32:44.175876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.175853    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-openshift-ca\") pod \"6585fca7-1520-47ab-ad9b-fb683ef63e67\" (UID: \"6585fca7-1520-47ab-ad9b-fb683ef63e67\") "
Nov 02 15:32:44.176498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.176341    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-openshift-ca" pod "6585fca7-1520-47ab-ad9b-fb683ef63e67" (UID: "6585fca7-1520-47ab-ad9b-fb683ef63e67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:44.177444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.177421    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:44.183625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.183599    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6585fca7-1520-47ab-ad9b-fb683ef63e67-kube-api-access-fzwrg" pod "6585fca7-1520-47ab-ad9b-fb683ef63e67" (UID: "6585fca7-1520-47ab-ad9b-fb683ef63e67"). InnerVolumeSpecName "kube-api-access-fzwrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:44.184768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.184741    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-xks-ca" pod "6585fca7-1520-47ab-ad9b-fb683ef63e67" (UID: "6585fca7-1520-47ab-ad9b-fb683ef63e67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:32:44.279116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.279082    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6585fca7-1520-47ab-ad9b-fb683ef63e67-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:44.279116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.279112    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzwrg\" (UniqueName: \"kubernetes.io/projected/6585fca7-1520-47ab-ad9b-fb683ef63e67-kube-api-access-fzwrg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:44.909906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.909871    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"
Nov 02 15:32:44.909906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.909890    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj" event={"ID":"6585fca7-1520-47ab-ad9b-fb683ef63e67","Type":"ContainerDied","Data":"b975d14687d6c9084015cb32e92b4fe9e85f06e7c93fc377bb5eee9d4a742845"}
Nov 02 15:32:44.910137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:44.909952    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b975d14687d6c9084015cb32e92b4fe9e85f06e7c93fc377bb5eee9d4a742845"
Nov 02 15:32:45.291616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.291514    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:45.651223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.651194    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:32:45.809263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809224    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-podinfo\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809280    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-certs\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809370    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b13b6c4-d185-42a0-bc81-0a8461455755-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13b6c4-d185-42a0-bc81-0a8461455755-istiod-ca-cert\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809457    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-token\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-token\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809594    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-data\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809671    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-kube-api-access-jhvpf\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-kube-api-access-jhvpf\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809678    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-certs" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.809722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809714    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-envoy\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809756    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-socket\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809785    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b13b6c4-d185-42a0-bc81-0a8461455755-istiod-ca-cert" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:32:45.809970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809796    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-credential-socket\") pod \"9b13b6c4-d185-42a0-bc81-0a8461455755\" (UID: \"9b13b6c4-d185-42a0-bc81-0a8461455755\") "
Nov 02 15:32:45.809970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.809895    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-data" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.810185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.810069    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-socket" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.810247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.810235    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-credential-socket" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.810665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.810628    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.810803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.810689    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.810803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.810704    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.810803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.810717    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b13b6c4-d185-42a0-bc81-0a8461455755-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.810803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.810730    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.812283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.812257    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-kube-api-access-jhvpf" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "kube-api-access-jhvpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:45.812732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.812705    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-envoy" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:32:45.812836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.812793    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-podinfo" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:32:45.812885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.812841    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-token" pod "9b13b6c4-d185-42a0-bc81-0a8461455755" (UID: "9b13b6c4-d185-42a0-bc81-0a8461455755"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:32:45.912570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.912537    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvpf\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-kube-api-access-jhvpf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.912570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.912567    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.912928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.912583    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.912928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.912591    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b13b6c4-d185-42a0-bc81-0a8461455755-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:32:45.921137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.921107    7019 generic.go:356] "Generic (PLEG): container finished" podID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerID="c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096" exitCode=0
Nov 02 15:32:45.921298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.921168    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"
Nov 02 15:32:45.921298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.921196    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" event={"ID":"9b13b6c4-d185-42a0-bc81-0a8461455755","Type":"ContainerDied","Data":"c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096"}
Nov 02 15:32:45.921298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.921255    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl" event={"ID":"9b13b6c4-d185-42a0-bc81-0a8461455755","Type":"ContainerDied","Data":"188a2ff0457fdb69e3dabb0e032606a78df9582790b6084181b53bc0355b3c51"}
Nov 02 15:32:45.921298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.921279    7019 scope.go:122] "RemoveContainer" containerID="c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096"
Nov 02 15:32:45.932641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.932623    7019 scope.go:122] "RemoveContainer" containerID="c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096"
Nov 02 15:32:45.932945 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:32:45.932922    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096\": container with ID starting with c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096 not found: ID does not exist" containerID="c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096"
Nov 02 15:32:45.933027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.932951    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096"} err="failed to get container status \"c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096\": rpc error: code = NotFound desc = could not find container \"c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096\": container with ID starting with c79208737a50eafb2586f5b70f97088568c7fb417ccdd03867a25321c5758096 not found: ID does not exist"
Nov 02 15:32:45.981004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.980956    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"]
Nov 02 15:32:45.985174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:45.985143    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a03e84-openshift-default-8f4cc949-ttgsl"]
Nov 02 15:32:46.886732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:46.886694    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:47.869611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:47.869569    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" path="/var/lib/kubelet/pods/9b13b6c4-d185-42a0-bc81-0a8461455755/volumes"
Nov 02 15:32:48.577208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:48.577164    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:48.730233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:48.730195    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:50.459280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:50.459241    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:51.578464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:51.578425    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:51.731110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:51.731070    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:52.103047 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:52.103008    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:52.708401 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:32:52.708348    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:32:53.712788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:53.712734    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:55.336227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:55.336192    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:57.022188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:57.022092    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:58.592114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:58.592038    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:32:59.985628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:32:59.985592    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:00.622733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:00.618864    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:00.656616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:00.656580    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"]
Nov 02 15:33:00.669798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:00.669710    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2abeaa-thdmt"]
Nov 02 15:33:00.776704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:00.776614    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:01.192468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:01.192423    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:01.865611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:01.865572    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8de6019c-106f-4015-9876-b8b4ba2de54b" path="/var/lib/kubelet/pods/8de6019c-106f-4015-9876-b8b4ba2de54b/volumes"
Nov 02 15:33:01.991344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:01.991311    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:02.437176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:02.437132    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:04.040199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:04.040169    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:05.592823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:05.592789    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:07.395548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:07.395503    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:07.894115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:07.894076    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:08.046979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:08.046944    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:08.238062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:08.237969    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"]
Nov 02 15:33:08.238235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:08.238160    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy" containerID="cri-o://4442e293b635cfb1dfd20386a7f9d3165007c729b42a2f22368e146db7f6a02e" gracePeriod=30
Nov 02 15:33:08.302160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:08.302124    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:08.833305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:08.833267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:09.240054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:09.239974    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:10.110948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:10.110899    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:11.081382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:11.081349    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:12.697139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:12.697095    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:13.334976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:13.334940    7019 generic.go:356] "Generic (PLEG): container finished" podID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerID="4442e293b635cfb1dfd20386a7f9d3165007c729b42a2f22368e146db7f6a02e" exitCode=0
Nov 02 15:33:13.335154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:13.335016    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" event={"ID":"fa84d2fd-0959-4ef0-baf5-d59c36f24337","Type":"ContainerDied","Data":"4442e293b635cfb1dfd20386a7f9d3165007c729b42a2f22368e146db7f6a02e"}
Nov 02 15:33:13.335154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:13.335096    7019 scope.go:122] "RemoveContainer" containerID="1a71d7181042133f56566a3eebc3b3ba503559008119cafb0993a645a0cad232"
Nov 02 15:33:13.943299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:13.943271    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:33:14.063785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.063686    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-kube-api-access-jdvv9\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-kube-api-access-jdvv9\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.063785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.063754    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-credential-socket\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.064046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.063832    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-podinfo\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.064046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.063878    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-certs\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.064046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.063995    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-socket\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.064221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.064057    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-data\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.064221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.064149    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-token\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-token\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.064221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.064176    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istiod-ca-cert\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.064442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.064281    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-envoy\") pod \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\" (UID: \"fa84d2fd-0959-4ef0-baf5-d59c36f24337\") "
Nov 02 15:33:14.065259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.065007    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-certs" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:14.065259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.065036    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istiod-ca-cert" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:14.065259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.065229    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-socket" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:14.065584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.065553    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-credential-socket" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:14.066954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.066924    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-podinfo" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:33:14.067247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.067222    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-data" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:14.083749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.077080    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-envoy" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:14.085213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.085162    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-kube-api-access-jdvv9" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "kube-api-access-jdvv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:14.089504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.089474    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-token" pod "fa84d2fd-0959-4ef0-baf5-d59c36f24337" (UID: "fa84d2fd-0959-4ef0-baf5-d59c36f24337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171025    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171068    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171083    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171096    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvv9\" (UniqueName: \"kubernetes.io/projected/fa84d2fd-0959-4ef0-baf5-d59c36f24337-kube-api-access-jdvv9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171112    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171128    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171142    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171155    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.172322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.171168    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa84d2fd-0959-4ef0-baf5-d59c36f24337-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:14.345898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.345806    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl" event={"ID":"fa84d2fd-0959-4ef0-baf5-d59c36f24337","Type":"ContainerDied","Data":"339f07281a368f967621748f8c4ee124d43b87339c33490e9bcef355d4c9393f"}
Nov 02 15:33:14.345898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.345837    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"
Nov 02 15:33:14.345898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.345864    7019 scope.go:122] "RemoveContainer" containerID="4442e293b635cfb1dfd20386a7f9d3165007c729b42a2f22368e146db7f6a02e"
Nov 02 15:33:14.374211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.374168    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:14.424045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.422570    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"]
Nov 02 15:33:14.431263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:14.431217    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2abeaa-openshift-default-ccdd5d949-nhltl"]
Nov 02 15:33:15.871790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:15.871746    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" path="/var/lib/kubelet/pods/fa84d2fd-0959-4ef0-baf5-d59c36f24337/volumes"
Nov 02 15:33:15.912672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:15.912607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:17.574628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:17.574587    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:19.276586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:19.276550    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:20.828876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:20.828839    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:22.510153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:22.510112    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:22.707381 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:33:22.707341    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:33:24.097888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:24.097847    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:25.588940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:25.588894    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:27.202848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:27.202813    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:28.886673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:28.884527    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:30.206267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:30.205410    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:30.752673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:30.734223    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:30.868696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:30.868205    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"]
Nov 02 15:33:30.874414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:30.874383    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3daba5-zl8qj"]
Nov 02 15:33:30.895432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:30.895352    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:31.049868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:31.049762    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:31.504993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:31.504957    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:31.863888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:31.863857    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6585fca7-1520-47ab-ad9b-fb683ef63e67" path="/var/lib/kubelet/pods/6585fca7-1520-47ab-ad9b-fb683ef63e67/volumes"
Nov 02 15:33:32.828901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:32.828765    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:34.431859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:34.431811    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:35.995492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:35.995443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:37.658459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:37.658423    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:37.883785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:37.883749    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:38.031848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:38.031765    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:38.141505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:38.141466    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"]
Nov 02 15:33:38.141752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:38.141713    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" containerID="cri-o://fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0" gracePeriod=30
Nov 02 15:33:38.205783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:38.205748    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:39.376922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:39.376885    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:40.498266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:40.498225    7019 patch_prober.go:29] interesting pod/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:33:40.498702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:40.498291    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:33:41.021196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:41.021155    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:42.588803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:42.588761    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:43.450455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.450430    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:33:43.599821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.599783    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-podinfo\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.599893    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-envoy\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.599967    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-socket\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600094    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-token\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-token\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600186    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-data\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600253    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-certs\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600496    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-certs" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.600619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600543    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-credential-socket\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600625    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-data" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.600755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600680    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istiod-ca-cert\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.600755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.600717    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-kube-api-access-7f9pt\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-kube-api-access-7f9pt\") pod \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\" (UID: \"92d092a7-8dbc-4d1f-a67f-ad9ca8645796\") "
Nov 02 15:33:43.601298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601089    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istiod-ca-cert" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:33:43.601298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601239    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-socket" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.601475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601362    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-credential-socket" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.601567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601547    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.601629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601572    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.601629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601587    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.601629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601600    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.601629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.601614    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.602627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.602598    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-envoy" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:33:43.604594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.604571    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-kube-api-access-7f9pt" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "kube-api-access-7f9pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:43.604594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.604580    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-podinfo" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:33:43.604760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.604596    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-token" pod "92d092a7-8dbc-4d1f-a67f-ad9ca8645796" (UID: "92d092a7-8dbc-4d1f-a67f-ad9ca8645796"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:33:43.649109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.649079    7019 generic.go:356] "Generic (PLEG): container finished" podID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerID="fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0" exitCode=0
Nov 02 15:33:43.649285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.649128    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"
Nov 02 15:33:43.649285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.649155    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" event={"ID":"92d092a7-8dbc-4d1f-a67f-ad9ca8645796","Type":"ContainerDied","Data":"fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0"}
Nov 02 15:33:43.649285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.649212    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr" event={"ID":"92d092a7-8dbc-4d1f-a67f-ad9ca8645796","Type":"ContainerDied","Data":"7c61733d73eea85fa855a9c49afe9d53ed2efe85643a2b17ea2c34567e9c1070"}
Nov 02 15:33:43.649285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.649239    7019 scope.go:122] "RemoveContainer" containerID="fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0"
Nov 02 15:33:43.662270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.662243    7019 scope.go:122] "RemoveContainer" containerID="bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984"
Nov 02 15:33:43.681335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.681207    7019 scope.go:122] "RemoveContainer" containerID="fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0"
Nov 02 15:33:43.682083 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:33:43.681907    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0\": container with ID starting with fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0 not found: ID does not exist" containerID="fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0"
Nov 02 15:33:43.682083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.681945    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0"} err="failed to get container status \"fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0\": rpc error: code = NotFound desc = could not find container \"fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0\": container with ID starting with fffcba838c8bf11e3407665eb695162ee3ebc6dbff60e7ca579aab403ac2a0f0 not found: ID does not exist"
Nov 02 15:33:43.682083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.681965    7019 scope.go:122] "RemoveContainer" containerID="bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984"
Nov 02 15:33:43.682338 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:33:43.682288    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984\": container with ID starting with bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984 not found: ID does not exist" containerID="bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984"
Nov 02 15:33:43.682338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.682320    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984"} err="failed to get container status \"bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984\": rpc error: code = NotFound desc = could not find container \"bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984\": container with ID starting with bd6044f99cb017c8f4a785327ec547fdf8b993315e105ab8a79051ec2738e984 not found: ID does not exist"
Nov 02 15:33:43.703107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.703075    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.703107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.703107    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.703322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.703122    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9pt\" (UniqueName: \"kubernetes.io/projected/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-kube-api-access-7f9pt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.703322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.703138    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d092a7-8dbc-4d1f-a67f-ad9ca8645796-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:33:43.707612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.707577    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"]
Nov 02 15:33:43.715916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.715878    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3daba5-openshift-default-59d44c74df-t9bbr"]
Nov 02 15:33:43.866052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:43.866007    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" path="/var/lib/kubelet/pods/92d092a7-8dbc-4d1f-a67f-ad9ca8645796/volumes"
Nov 02 15:33:44.005288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:44.005200    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:44.147476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:44.147432    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:45.704402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:45.704366    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:46.494082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:46.494032    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:47.508343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:47.508293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:48.659684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:48.659631    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:48.771044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:48.771013    7019 scope.go:122] "RemoveContainer" containerID="c81b6a6c4df93459a5bda1e200129f25c2efc964214c23413b4b545c9fc75b54"
Nov 02 15:33:48.783009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:48.782984    7019 scope.go:122] "RemoveContainer" containerID="d0ca25eea1e388ccce15e28211be1dd62cd76f1dd89927ee8f6e69f7c75b4f63"
Nov 02 15:33:48.792497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:48.792459    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:33:48.795444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:48.795421    7019 scope.go:122] "RemoveContainer" containerID="1086f9c471a3cfda49a9b6fbb26a94457613b7dd3915a3f7eb7a01562a539891"
Nov 02 15:33:49.038825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:49.038738    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:50.882673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:50.877323    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:52.490111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:52.490068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:52.712945 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:33:52.712890    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:33:54.126730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:54.126691    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:55.686795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:55.686758    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:57.374399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:57.374355    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:33:59.178253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:33:59.178215    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:00.750749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:00.750709    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:02.113340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:02.113303    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:02.373640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:02.373551    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:03.985266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:03.985228    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:05.680591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:05.680549    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:07.171761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:07.171723    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:08.870533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:08.870486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:10.492438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:10.492391    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:12.186295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:12.185919    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:13.710356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:13.710307    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:15.322815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:15.322768    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:17.004417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:17.004369    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:17.641463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:17.641422    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:18.677235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:18.677180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:20.431165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:20.431121    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:22.210193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:22.210154    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:22.708613 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:34:22.708570    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:34:23.813763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:23.810852    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:25.403692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:25.403379    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:27.109563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:27.109467    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:29.016187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:29.016150    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:30.641814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:30.641768    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:32.019539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:32.019502    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:32.401684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:32.395387    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:34.092604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:34.092564    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:35.770965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:35.770916    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:37.403159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:37.403110    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:39.001643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:39.001608    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:40.602484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:40.602443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:42.349416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:42.349365    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:44.033347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:44.033266    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:45.873178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:45.873141    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:47.522713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:47.522620    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:48.774775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:48.774734    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:48.856291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:48.856260    7019 scope.go:122] "RemoveContainer" containerID="4e29bfbd547b4658d0d2f76afdf2aded3803390999a6a549023abc8405aef219"
Nov 02 15:34:49.107821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:49.107735    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:50.695387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:50.695350    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:52.243117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:52.243081    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:52.709697 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:34:52.709642    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:34:53.893586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:53.893546    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:54.905937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:54.905893    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:55.674745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:55.674602    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:57.270541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:57.270506    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:34:58.898482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:34:58.898437    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:00.580424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:00.580383    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:02.261536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.261484    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:02.432210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.432171    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:02.974587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.974541    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"]
Nov 02 15:35:02.975703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975679    7019 cpu_manager.go:414] "containerMap: removing container" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy"
Nov 02 15:35:02.975781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975705    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy"
Nov 02 15:35:02.975781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975745    7019 cpu_manager.go:414] "containerMap: removing container" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy"
Nov 02 15:35:02.975781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975753    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy"
Nov 02 15:35:02.975781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975768    7019 cpu_manager.go:414] "containerMap: removing container" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy"
Nov 02 15:35:02.975781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975776    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy"
Nov 02 15:35:02.975958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975793    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy"
Nov 02 15:35:02.975958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975800    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy"
Nov 02 15:35:02.975958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975814    7019 cpu_manager.go:414] "containerMap: removing container" podUID="6585fca7-1520-47ab-ad9b-fb683ef63e67" containerName="revoke-keys"
Nov 02 15:35:02.975958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975821    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6585fca7-1520-47ab-ad9b-fb683ef63e67" containerName="revoke-keys"
Nov 02 15:35:02.975958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975835    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy"
Nov 02 15:35:02.975958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.975842    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy"
Nov 02 15:35:02.976140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.976021    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy"
Nov 02 15:35:02.976140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.976034    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b13b6c4-d185-42a0-bc81-0a8461455755" containerName="istio-proxy"
Nov 02 15:35:02.976140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.976047    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d092a7-8dbc-4d1f-a67f-ad9ca8645796" containerName="istio-proxy"
Nov 02 15:35:02.976140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.976055    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy"
Nov 02 15:35:02.976140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.976063    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6585fca7-1520-47ab-ad9b-fb683ef63e67" containerName="revoke-keys"
Nov 02 15:35:02.976140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.976074    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa84d2fd-0959-4ef0-baf5-d59c36f24337" containerName="istio-proxy"
Nov 02 15:35:02.982203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.982177    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:02.998235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:02.998159    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"]
Nov 02 15:35:03.011695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.008983    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:35:03.011695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.009056    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:35:03.123581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.123539    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq28d\" (UniqueName: \"kubernetes.io/projected/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-kube-api-access-fq28d\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.123833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.123814    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.123913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.123890    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.225955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.225862    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.226124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.225972    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.226124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.226061    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq28d\" (UniqueName: \"kubernetes.io/projected/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-kube-api-access-fq28d\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.226757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.226733    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.228381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.228356    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.251271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.251236    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq28d\" (UniqueName: \"kubernetes.io/projected/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-kube-api-access-fq28d\") pod \"maas-api-revoke-sub-7fcf0be2-9qqj2\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.309114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.309075    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:03.642596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.642565    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"]
Nov 02 15:35:03.642775 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:35:03.642751    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc2fac3_fc3c_4cc7_ac50_42a48009ceb0.slice/crio-7c00ddf5bd249fd75e7e2a6ed73dc0cc865f9868b757e305185e44fde570bead WatchSource:0}: Error finding container 7c00ddf5bd249fd75e7e2a6ed73dc0cc865f9868b757e305185e44fde570bead: Status 404 returned error can't find the container with id 7c00ddf5bd249fd75e7e2a6ed73dc0cc865f9868b757e305185e44fde570bead
Nov 02 15:35:03.921452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:03.921366    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:04.652376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:04.652336    7019 generic.go:356] "Generic (PLEG): container finished" podID="8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" containerID="fca7c47491d222d846931efcbb7579d08751e53c00c9bbdcd40a10ec501015fd" exitCode=0
Nov 02 15:35:04.652823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:04.652416    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2" event={"ID":"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0","Type":"ContainerDied","Data":"fca7c47491d222d846931efcbb7579d08751e53c00c9bbdcd40a10ec501015fd"}
Nov 02 15:35:04.652823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:04.652470    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2" event={"ID":"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0","Type":"ContainerStarted","Data":"7c00ddf5bd249fd75e7e2a6ed73dc0cc865f9868b757e305185e44fde570bead"}
Nov 02 15:35:05.731152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.730719    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:05.875360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.875330    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:05.994979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.994885    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-openshift-ca\") pod \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") "
Nov 02 15:35:05.995164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.995035    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-xks-ca\") pod \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") "
Nov 02 15:35:05.995236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.995222    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-kube-api-access-fq28d\" (UniqueName: \"kubernetes.io/projected/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-kube-api-access-fq28d\") pod \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\" (UID: \"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0\") "
Nov 02 15:35:05.995301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.995245    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-openshift-ca" pod "8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" (UID: "8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:05.996753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.996178    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:05.998814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.998785    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-xks-ca" pod "8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" (UID: "8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:05.998940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:05.998859    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-kube-api-access-fq28d" pod "8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" (UID: "8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0"). InnerVolumeSpecName "kube-api-access-fq28d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:06.097626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:06.097587    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq28d\" (UniqueName: \"kubernetes.io/projected/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-kube-api-access-fq28d\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:06.097626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:06.097619    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:06.671682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:06.671616    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"
Nov 02 15:35:06.671682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:06.671639    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2" event={"ID":"8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0","Type":"ContainerDied","Data":"7c00ddf5bd249fd75e7e2a6ed73dc0cc865f9868b757e305185e44fde570bead"}
Nov 02 15:35:06.671919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:06.671728    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c00ddf5bd249fd75e7e2a6ed73dc0cc865f9868b757e305185e44fde570bead"
Nov 02 15:35:07.427670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:07.417193    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:07.683945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:07.683857    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:09.000289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:09.000253    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:10.593990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:10.593919    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:12.250295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:12.250256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:13.840235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:13.840194    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"]
Nov 02 15:35:13.899775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:13.899726    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:14.060678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.055543    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:14.376531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.376494    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"]
Nov 02 15:35:14.377328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.377310    7019 cpu_manager.go:414] "containerMap: removing container" podUID="8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" containerName="revoke-keys"
Nov 02 15:35:14.377378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.377330    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" containerName="revoke-keys"
Nov 02 15:35:14.377461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.377451    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" containerName="revoke-keys"
Nov 02 15:35:14.381183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.381148    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.398518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.398257    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:35:14.398838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.398820    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:35:14.404793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.404764    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"]
Nov 02 15:35:14.433307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.433278    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmsq\" (UniqueName: \"kubernetes.io/projected/06357e76-a190-4dba-9cbc-4a6b32662f89-kube-api-access-6gmsq\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.433580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.433552    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.433697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.433670    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.535153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.535112    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.535364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.535201    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.535364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.535344    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmsq\" (UniqueName: \"kubernetes.io/projected/06357e76-a190-4dba-9cbc-4a6b32662f89-kube-api-access-6gmsq\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.535986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.535954    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.537584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.537563    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.559576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.559550    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmsq\" (UniqueName: \"kubernetes.io/projected/06357e76-a190-4dba-9cbc-4a6b32662f89-kube-api-access-6gmsq\") pod \"maas-api-revoke-sub-96d9d05a-w6fw7\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:14.701003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:14.700915    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:15.013620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:15.013589    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"]
Nov 02 15:35:15.594327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:15.594292    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:15.764662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:15.764548    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7" event={"ID":"06357e76-a190-4dba-9cbc-4a6b32662f89","Type":"ContainerStarted","Data":"810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40"}
Nov 02 15:35:15.764662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:15.764607    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7" event={"ID":"06357e76-a190-4dba-9cbc-4a6b32662f89","Type":"ContainerStarted","Data":"299061c7c29ffe38840f0db22aa54899d8d0c6d127f1b6399c7857c06fad89dc"}
Nov 02 15:35:15.812330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:15.812266    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7" podStartSLOduration=1.8122527480000001 podStartE2EDuration="1.812252748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:15.80858519 +0000 UTC m=+2310.062538821" watchObservedRunningTime="2026-11-02 15:35:15.812252748 +0000 UTC m=+2310.066206372"
Nov 02 15:35:16.796675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:16.796616    7019 generic.go:356] "Generic (PLEG): container finished" podID="06357e76-a190-4dba-9cbc-4a6b32662f89" containerID="810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40" exitCode=0
Nov 02 15:35:16.797155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:16.796912    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7" event={"ID":"06357e76-a190-4dba-9cbc-4a6b32662f89","Type":"ContainerDied","Data":"810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40"}
Nov 02 15:35:17.409541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:17.409504    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:17.973533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:17.973501    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:18.110013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.109907    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06357e76-a190-4dba-9cbc-4a6b32662f89-kube-api-access-6gmsq\" (UniqueName: \"kubernetes.io/projected/06357e76-a190-4dba-9cbc-4a6b32662f89-kube-api-access-6gmsq\") pod \"06357e76-a190-4dba-9cbc-4a6b32662f89\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") "
Nov 02 15:35:18.110013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.110001    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-xks-ca\") pod \"06357e76-a190-4dba-9cbc-4a6b32662f89\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") "
Nov 02 15:35:18.110254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.110072    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-openshift-ca\") pod \"06357e76-a190-4dba-9cbc-4a6b32662f89\" (UID: \"06357e76-a190-4dba-9cbc-4a6b32662f89\") "
Nov 02 15:35:18.110471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.110443    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-openshift-ca" pod "06357e76-a190-4dba-9cbc-4a6b32662f89" (UID: "06357e76-a190-4dba-9cbc-4a6b32662f89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:18.110986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.110960    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:18.112351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.112326    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-xks-ca" pod "06357e76-a190-4dba-9cbc-4a6b32662f89" (UID: "06357e76-a190-4dba-9cbc-4a6b32662f89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:18.112574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.112555    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06357e76-a190-4dba-9cbc-4a6b32662f89-kube-api-access-6gmsq" pod "06357e76-a190-4dba-9cbc-4a6b32662f89" (UID: "06357e76-a190-4dba-9cbc-4a6b32662f89"). InnerVolumeSpecName "kube-api-access-6gmsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:18.213262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.213217    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gmsq\" (UniqueName: \"kubernetes.io/projected/06357e76-a190-4dba-9cbc-4a6b32662f89-kube-api-access-6gmsq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:18.213262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.213258    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06357e76-a190-4dba-9cbc-4a6b32662f89-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:18.817199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.817163    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"
Nov 02 15:35:18.817199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.817188    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7" event={"ID":"06357e76-a190-4dba-9cbc-4a6b32662f89","Type":"ContainerDied","Data":"299061c7c29ffe38840f0db22aa54899d8d0c6d127f1b6399c7857c06fad89dc"}
Nov 02 15:35:18.817472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:18.817238    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="299061c7c29ffe38840f0db22aa54899d8d0c6d127f1b6399c7857c06fad89dc"
Nov 02 15:35:19.100405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:19.100323    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:19.735279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:19.735244    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:35:20.746670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:20.746068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:22.609774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:22.601483    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:22.708539 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:35:22.708503    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:35:22.957488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:22.957446    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:24.394543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:24.394499    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:26.099850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:26.099805    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:27.783784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:27.783744    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:28.877705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:28.877632    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh" podUID="4a86c4cc-bc5e-4b27-b461-e5a590237005" containerName="main" containerID="cri-o://8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4" gracePeriod=45
Nov 02 15:35:29.169628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.169598    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:35:29.225282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.225244    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-tmp-dir\") pod \"4a86c4cc-bc5e-4b27-b461-e5a590237005\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") "
Nov 02 15:35:29.225495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.225303    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a86c4cc-bc5e-4b27-b461-e5a590237005-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86c4cc-bc5e-4b27-b461-e5a590237005-tls-certs\") pod \"4a86c4cc-bc5e-4b27-b461-e5a590237005\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") "
Nov 02 15:35:29.225495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.225490    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-dshm\") pod \"4a86c4cc-bc5e-4b27-b461-e5a590237005\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") "
Nov 02 15:35:29.225632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.225565    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-home\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-home\") pod \"4a86c4cc-bc5e-4b27-b461-e5a590237005\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") "
Nov 02 15:35:29.225776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.225751    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a86c4cc-bc5e-4b27-b461-e5a590237005-kube-api-access-bh42s\" (UniqueName: \"kubernetes.io/projected/4a86c4cc-bc5e-4b27-b461-e5a590237005-kube-api-access-bh42s\") pod \"4a86c4cc-bc5e-4b27-b461-e5a590237005\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") "
Nov 02 15:35:29.226001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.225973    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-model-cache\") pod \"4a86c4cc-bc5e-4b27-b461-e5a590237005\" (UID: \"4a86c4cc-bc5e-4b27-b461-e5a590237005\") "
Nov 02 15:35:29.227525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.227247    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-model-cache" pod "4a86c4cc-bc5e-4b27-b461-e5a590237005" (UID: "4a86c4cc-bc5e-4b27-b461-e5a590237005"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:29.227920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.227872    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-home" pod "4a86c4cc-bc5e-4b27-b461-e5a590237005" (UID: "4a86c4cc-bc5e-4b27-b461-e5a590237005"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:29.228306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.228276    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-tmp-dir" pod "4a86c4cc-bc5e-4b27-b461-e5a590237005" (UID: "4a86c4cc-bc5e-4b27-b461-e5a590237005"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:29.231241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.231170    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a86c4cc-bc5e-4b27-b461-e5a590237005-kube-api-access-bh42s" pod "4a86c4cc-bc5e-4b27-b461-e5a590237005" (UID: "4a86c4cc-bc5e-4b27-b461-e5a590237005"). InnerVolumeSpecName "kube-api-access-bh42s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:29.238064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.238012    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a86c4cc-bc5e-4b27-b461-e5a590237005-tls-certs" pod "4a86c4cc-bc5e-4b27-b461-e5a590237005" (UID: "4a86c4cc-bc5e-4b27-b461-e5a590237005"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:29.248130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.248101    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-dshm" pod "4a86c4cc-bc5e-4b27-b461-e5a590237005" (UID: "4a86c4cc-bc5e-4b27-b461-e5a590237005"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:35:29.328619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.328588    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:29.328619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.328617    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:29.328619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.328626    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a86c4cc-bc5e-4b27-b461-e5a590237005-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:29.328944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.328634    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:29.328944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.328642    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a86c4cc-bc5e-4b27-b461-e5a590237005-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:29.328944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.328681    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh42s\" (UniqueName: \"kubernetes.io/projected/4a86c4cc-bc5e-4b27-b461-e5a590237005-kube-api-access-bh42s\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:29.535743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.533716    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:29.930885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.930847    7019 generic.go:356] "Generic (PLEG): container finished" podID="4a86c4cc-bc5e-4b27-b461-e5a590237005" containerID="8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4" exitCode=0
Nov 02 15:35:29.931295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.930900    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"
Nov 02 15:35:29.931295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.930942    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh" event={"ID":"4a86c4cc-bc5e-4b27-b461-e5a590237005","Type":"ContainerDied","Data":"8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4"}
Nov 02 15:35:29.931295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.930981    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh" event={"ID":"4a86c4cc-bc5e-4b27-b461-e5a590237005","Type":"ContainerDied","Data":"5dac7f2a87180d1feec7d82ebfb58740ed68158f85dceff57eedb7e3ad4e718d"}
Nov 02 15:35:29.931295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.931000    7019 scope.go:122] "RemoveContainer" containerID="8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4"
Nov 02 15:35:29.942245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.942219    7019 scope.go:122] "RemoveContainer" containerID="8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4"
Nov 02 15:35:29.942592 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:35:29.942565    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4\": container with ID starting with 8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4 not found: ID does not exist" containerID="8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4"
Nov 02 15:35:29.942686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:29.942607    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4"} err="failed to get container status \"8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4\": rpc error: code = NotFound desc = could not find container \"8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4\": container with ID starting with 8afce90bd27df6f8975efa109fcf6379142b7c7eacc016cfbaa790c09acfeeb4 not found: ID does not exist"
Nov 02 15:35:30.006118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:30.006090    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"]
Nov 02 15:35:30.019073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:30.019040    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-28f621/test-model-28f621-kserve-69984cddd9-zmtlh"]
Nov 02 15:35:31.323704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:31.322859    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:31.863938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:31.863906    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a86c4cc-bc5e-4b27-b461-e5a590237005" path="/var/lib/kubelet/pods/4a86c4cc-bc5e-4b27-b461-e5a590237005/volumes"
Nov 02 15:35:32.870346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:32.870313    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:34.591170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:34.591098    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:36.223485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:36.223448    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:37.909238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:37.909196    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:39.626140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:39.626068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:41.299616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:41.299579    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:42.841327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:42.841291    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:43.307200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.307161    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"]
Nov 02 15:35:43.308260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.308238    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4a86c4cc-bc5e-4b27-b461-e5a590237005" containerName="main"
Nov 02 15:35:43.308329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.308263    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a86c4cc-bc5e-4b27-b461-e5a590237005" containerName="main"
Nov 02 15:35:43.308368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.308336    7019 cpu_manager.go:414] "containerMap: removing container" podUID="06357e76-a190-4dba-9cbc-4a6b32662f89" containerName="revoke-keys"
Nov 02 15:35:43.308368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.308346    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06357e76-a190-4dba-9cbc-4a6b32662f89" containerName="revoke-keys"
Nov 02 15:35:43.308549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.308536    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="06357e76-a190-4dba-9cbc-4a6b32662f89" containerName="revoke-keys"
Nov 02 15:35:43.308588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.308560    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a86c4cc-bc5e-4b27-b461-e5a590237005" containerName="main"
Nov 02 15:35:43.313148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.313126    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.330594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.330566    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"]
Nov 02 15:35:43.339416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.339302    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:35:43.339416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.339399    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:35:43.361436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.361396    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.361871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.361834    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlnw\" (UniqueName: \"kubernetes.io/projected/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-kube-api-access-6jlnw\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.361998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.361929    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.463792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.463760    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.463955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.463876    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlnw\" (UniqueName: \"kubernetes.io/projected/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-kube-api-access-6jlnw\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.463955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.463921    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.464391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.464370    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.466297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.466266    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.490174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.490144    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlnw\" (UniqueName: \"kubernetes.io/projected/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-kube-api-access-6jlnw\") pod \"maas-api-revoke-sub-45a38853-pw95j\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.629288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.629258    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:43.978301 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:35:43.978268    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0416c2b7_415d_4c4e_9f9f_bfe84ef19fc3.slice/crio-39e75ed2fa6704362eab78bac80ffa069dccd2b98128e695338c45f6e4910186 WatchSource:0}: Error finding container 39e75ed2fa6704362eab78bac80ffa069dccd2b98128e695338c45f6e4910186: Status 404 returned error can't find the container with id 39e75ed2fa6704362eab78bac80ffa069dccd2b98128e695338c45f6e4910186
Nov 02 15:35:43.978746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:43.978364    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"]
Nov 02 15:35:44.062402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:44.062360    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j" event={"ID":"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3","Type":"ContainerStarted","Data":"39e75ed2fa6704362eab78bac80ffa069dccd2b98128e695338c45f6e4910186"}
Nov 02 15:35:44.594794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:44.594755    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:45.075777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:45.075737    7019 generic.go:356] "Generic (PLEG): container finished" podID="0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" containerID="b8647cffd623a0cf45312fae714ee8aea8ec440ecea16d7aa25f1fbd1a5e0cea" exitCode=0
Nov 02 15:35:45.076232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:45.075825    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j" event={"ID":"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3","Type":"ContainerDied","Data":"b8647cffd623a0cf45312fae714ee8aea8ec440ecea16d7aa25f1fbd1a5e0cea"}
Nov 02 15:35:46.266486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.266454    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:46.315255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.315223    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-xks-ca\") pod \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") "
Nov 02 15:35:46.315470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.315342    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-openshift-ca\") pod \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") "
Nov 02 15:35:46.315543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.315512    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-kube-api-access-6jlnw\" (UniqueName: \"kubernetes.io/projected/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-kube-api-access-6jlnw\") pod \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\" (UID: \"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3\") "
Nov 02 15:35:46.315827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.315790    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-openshift-ca" pod "0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" (UID: "0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:46.316468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.316437    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:46.317666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.317618    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-xks-ca" pod "0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" (UID: "0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:46.318029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.318000    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-kube-api-access-6jlnw" pod "0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" (UID: "0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3"). InnerVolumeSpecName "kube-api-access-6jlnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:46.401849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.401771    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:46.426678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.422428    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jlnw\" (UniqueName: \"kubernetes.io/projected/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-kube-api-access-6jlnw\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:46.426678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:46.422479    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:47.096900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.096861    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"
Nov 02 15:35:47.096900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.096888    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j" event={"ID":"0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3","Type":"ContainerDied","Data":"39e75ed2fa6704362eab78bac80ffa069dccd2b98128e695338c45f6e4910186"}
Nov 02 15:35:47.097197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.096949    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39e75ed2fa6704362eab78bac80ffa069dccd2b98128e695338c45f6e4910186"
Nov 02 15:35:47.570543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.570502    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"]
Nov 02 15:35:47.571283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.571266    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" containerName="revoke-keys"
Nov 02 15:35:47.571283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.571284    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" containerName="revoke-keys"
Nov 02 15:35:47.571413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.571398    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" containerName="revoke-keys"
Nov 02 15:35:47.575386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.575368    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.596372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.596311    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"]
Nov 02 15:35:47.607326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.602950    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:35:47.607326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.603178    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:35:47.641851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.641546    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.641851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.641802    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmc5x\" (UniqueName: \"kubernetes.io/projected/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-kube-api-access-dmc5x\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.641851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.641855    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.746292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.746245    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.746512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.746492    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmc5x\" (UniqueName: \"kubernetes.io/projected/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-kube-api-access-dmc5x\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.746576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.746565    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.747275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.747249    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.754124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.754096    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.771821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.771793    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmc5x\" (UniqueName: \"kubernetes.io/projected/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-kube-api-access-dmc5x\") pod \"maas-api-revoke-sub-1e564c91-vbt84\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:47.892218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:47.892182    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:48.043758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.027205    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:48.207116 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:35:48.207084    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf9945f_a963_4e3b_b40b_bdcc5d117f0e.slice/crio-edca7eb28db72f1ef6870a9538695582ca8d86049b071c84821df08a4e59962b WatchSource:0}: Error finding container edca7eb28db72f1ef6870a9538695582ca8d86049b071c84821df08a4e59962b: Status 404 returned error can't find the container with id edca7eb28db72f1ef6870a9538695582ca8d86049b071c84821df08a4e59962b
Nov 02 15:35:48.210217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.210186    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"]
Nov 02 15:35:48.308318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.308276    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"]
Nov 02 15:35:48.428667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.424088    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:48.609772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.609737    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"]
Nov 02 15:35:48.616827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.616802    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.634982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.634950    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"]
Nov 02 15:35:48.681723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.681688    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.681871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.681750    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpbp\" (UniqueName: \"kubernetes.io/projected/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-kube-api-access-vzpbp\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.682095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.682070    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.784213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.784116    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.784390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.784251    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.784390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.784288    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpbp\" (UniqueName: \"kubernetes.io/projected/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-kube-api-access-vzpbp\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.785200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.785078    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.786768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.786740    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.815442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.815407    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpbp\" (UniqueName: \"kubernetes.io/projected/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-kube-api-access-vzpbp\") pod \"maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.935465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.935432    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:48.963748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:48.963633    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:49.121536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:49.121501    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84" event={"ID":"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e","Type":"ContainerStarted","Data":"b807e1b17a320ea0ac2628e27ea9ba132553ebd98c53bc1e89f11f3bd9e9bfa5"}
Nov 02 15:35:49.121749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:49.121564    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84" event={"ID":"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e","Type":"ContainerStarted","Data":"edca7eb28db72f1ef6870a9538695582ca8d86049b071c84821df08a4e59962b"}
Nov 02 15:35:49.258806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:49.258759    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84" podStartSLOduration=2.258743875 podStartE2EDuration="2.258743875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:49.174059557 +0000 UTC m=+2343.428013204" watchObservedRunningTime="2026-11-02 15:35:49.258743875 +0000 UTC m=+2343.512697501"
Nov 02 15:35:49.259111 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:35:49.259089    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c707aa_4f71_4f35_bfc7_0fc945e49a7f.slice/crio-7b5151c7b83101fce94570fdd1212c76e2f18f5335832c8504f36f93c7cb7dca WatchSource:0}: Error finding container 7b5151c7b83101fce94570fdd1212c76e2f18f5335832c8504f36f93c7cb7dca: Status 404 returned error can't find the container with id 7b5151c7b83101fce94570fdd1212c76e2f18f5335832c8504f36f93c7cb7dca
Nov 02 15:35:49.260361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:49.260339    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"]
Nov 02 15:35:49.986214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:49.986169    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:50.130860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:50.130827    7019 generic.go:356] "Generic (PLEG): container finished" podID="bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" containerID="b807e1b17a320ea0ac2628e27ea9ba132553ebd98c53bc1e89f11f3bd9e9bfa5" exitCode=0
Nov 02 15:35:50.131082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:50.130905    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84" event={"ID":"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e","Type":"ContainerDied","Data":"b807e1b17a320ea0ac2628e27ea9ba132553ebd98c53bc1e89f11f3bd9e9bfa5"}
Nov 02 15:35:50.132661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:50.132624    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s" event={"ID":"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f","Type":"ContainerStarted","Data":"6ac344f23baefb76ab41efaca84784cbaf0fbc2d3a6332a7e52cc5c3d281c631"}
Nov 02 15:35:50.132799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:50.132694    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s" event={"ID":"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f","Type":"ContainerStarted","Data":"7b5151c7b83101fce94570fdd1212c76e2f18f5335832c8504f36f93c7cb7dca"}
Nov 02 15:35:50.236280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:50.236219    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s" podStartSLOduration=2.236199931 podStartE2EDuration="2.236199931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:35:50.230693509 +0000 UTC m=+2344.484647130" watchObservedRunningTime="2026-11-02 15:35:50.236199931 +0000 UTC m=+2344.490153558"
Nov 02 15:35:50.355291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:50.355258    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:51.142308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.142274    7019 generic.go:356] "Generic (PLEG): container finished" podID="d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" containerID="6ac344f23baefb76ab41efaca84784cbaf0fbc2d3a6332a7e52cc5c3d281c631" exitCode=0
Nov 02 15:35:51.142812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.142360    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s" event={"ID":"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f","Type":"ContainerDied","Data":"6ac344f23baefb76ab41efaca84784cbaf0fbc2d3a6332a7e52cc5c3d281c631"}
Nov 02 15:35:51.344329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.344305    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:51.426768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.426687    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-xks-ca\") pod \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") "
Nov 02 15:35:51.426768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.426741    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-openshift-ca\") pod \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") "
Nov 02 15:35:51.426974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.426831    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-kube-api-access-dmc5x\" (UniqueName: \"kubernetes.io/projected/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-kube-api-access-dmc5x\") pod \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\" (UID: \"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e\") "
Nov 02 15:35:51.427113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.427088    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-openshift-ca" pod "bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" (UID: "bcf9945f-a963-4e3b-b40b-bdcc5d117f0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:51.427811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.427694    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:51.429060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.429029    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-kube-api-access-dmc5x" pod "bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" (UID: "bcf9945f-a963-4e3b-b40b-bdcc5d117f0e"). InnerVolumeSpecName "kube-api-access-dmc5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:51.429336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.429298    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-xks-ca" pod "bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" (UID: "bcf9945f-a963-4e3b-b40b-bdcc5d117f0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:51.529277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.529238    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:51.529277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.529272    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmc5x\" (UniqueName: \"kubernetes.io/projected/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e-kube-api-access-dmc5x\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:51.784325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:51.784238    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:52.153224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.153183    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84" event={"ID":"bcf9945f-a963-4e3b-b40b-bdcc5d117f0e","Type":"ContainerDied","Data":"edca7eb28db72f1ef6870a9538695582ca8d86049b071c84821df08a4e59962b"}
Nov 02 15:35:52.153224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.153210    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"
Nov 02 15:35:52.153759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.153281    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edca7eb28db72f1ef6870a9538695582ca8d86049b071c84821df08a4e59962b"
Nov 02 15:35:52.336121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.336090    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:52.451175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.451096    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-xks-ca\") pod \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") "
Nov 02 15:35:52.451340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.451288    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-openshift-ca\") pod \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") "
Nov 02 15:35:52.451388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.451348    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-kube-api-access-vzpbp\" (UniqueName: \"kubernetes.io/projected/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-kube-api-access-vzpbp\") pod \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\" (UID: \"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f\") "
Nov 02 15:35:52.451712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.451680    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-openshift-ca" pod "d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" (UID: "d0c707aa-4f71-4f35-bfc7-0fc945e49a7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:35:52.452066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.452046    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:52.453170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.453146    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-xks-ca" pod "d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" (UID: "d0c707aa-4f71-4f35-bfc7-0fc945e49a7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:35:52.453479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.453460    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-kube-api-access-vzpbp" pod "d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" (UID: "d0c707aa-4f71-4f35-bfc7-0fc945e49a7f"). InnerVolumeSpecName "kube-api-access-vzpbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:35:52.553402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.553368    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzpbp\" (UniqueName: \"kubernetes.io/projected/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-kube-api-access-vzpbp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:52.553402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:52.553401    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:35:52.706410 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:35:52.706324    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:35:53.162785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:53.162754    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s" event={"ID":"d0c707aa-4f71-4f35-bfc7-0fc945e49a7f","Type":"ContainerDied","Data":"7b5151c7b83101fce94570fdd1212c76e2f18f5335832c8504f36f93c7cb7dca"}
Nov 02 15:35:53.162785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:53.162781    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"
Nov 02 15:35:53.163214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:53.162805    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b5151c7b83101fce94570fdd1212c76e2f18f5335832c8504f36f93c7cb7dca"
Nov 02 15:35:53.411600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:53.411552    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:55.081293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:55.081252    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"]
Nov 02 15:35:55.085669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:55.082062    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" containerID="cri-o://a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7" gracePeriod=30
Nov 02 15:35:55.488527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:55.488473    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:57.155492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:57.155387    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:58.798542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:58.798504    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:59.330934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:59.330897    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:59.471226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:59.471185    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:35:59.937907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:59.937873    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-tvp64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:35:59.938562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:35:59.938533    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:36:00.579740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.579683    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:36:00.678571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.678534    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00052150-74c8-4c65-8395-6b888102714b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00052150-74c8-4c65-8395-6b888102714b-istiod-ca-cert\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.678963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.678940    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-certs\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.679434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.679417    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-data\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.679705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.679686    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00052150-74c8-4c65-8395-6b888102714b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00052150-74c8-4c65-8395-6b888102714b-istio-podinfo\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.680328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.680308    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-envoy\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.680587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.680561    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-credential-socket\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.680777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.680761    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-socket\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.681130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.681111    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-istio-token\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-istio-token\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.681314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.681299    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-kube-api-access-pf47d\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-kube-api-access-pf47d\") pod \"00052150-74c8-4c65-8395-6b888102714b\" (UID: \"00052150-74c8-4c65-8395-6b888102714b\") "
Nov 02 15:36:00.683489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.679946    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00052150-74c8-4c65-8395-6b888102714b-istiod-ca-cert" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:00.683489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.680899    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-certs" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:00.683489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.681419    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-data" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:00.683963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.683944    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-credential-socket" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:00.684108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.684070    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-socket" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:00.693109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.692995    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:00.696902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.696864    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00052150-74c8-4c65-8395-6b888102714b-istio-podinfo" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:00.697855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.697826    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-istio-token" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:00.698087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.698073    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-kube-api-access-pf47d" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "kube-api-access-pf47d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:00.700038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.700011    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-envoy" pod "00052150-74c8-4c65-8395-6b888102714b" (UID: "00052150-74c8-4c65-8395-6b888102714b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784467    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00052150-74c8-4c65-8395-6b888102714b-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784506    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784518    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784543    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784555    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784568    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf47d\" (UniqueName: \"kubernetes.io/projected/00052150-74c8-4c65-8395-6b888102714b-kube-api-access-pf47d\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784580    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00052150-74c8-4c65-8395-6b888102714b-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784593    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:00.784709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:00.784604    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00052150-74c8-4c65-8395-6b888102714b-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:01.298084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.298047    7019 generic.go:356] "Generic (PLEG): container finished" podID="00052150-74c8-4c65-8395-6b888102714b" containerID="a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7" exitCode=0
Nov 02 15:36:01.298548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.298109    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"
Nov 02 15:36:01.298548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.298141    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" event={"ID":"00052150-74c8-4c65-8395-6b888102714b","Type":"ContainerDied","Data":"a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7"}
Nov 02 15:36:01.298548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.298205    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64" event={"ID":"00052150-74c8-4c65-8395-6b888102714b","Type":"ContainerDied","Data":"ccade901c9ae3b84070aa47e061e40d8a03e63f070dc9b02b1301e5866eee6cb"}
Nov 02 15:36:01.298548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.298235    7019 scope.go:122] "RemoveContainer" containerID="a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7"
Nov 02 15:36:01.310380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.310362    7019 scope.go:122] "RemoveContainer" containerID="a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7"
Nov 02 15:36:01.310680 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:36:01.310641    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7\": container with ID starting with a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7 not found: ID does not exist" containerID="a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7"
Nov 02 15:36:01.310757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.310692    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7"} err="failed to get container status \"a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7\": rpc error: code = NotFound desc = could not find container \"a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7\": container with ID starting with a26b9fe73d5a3f375d358c20550020c16a717ada463e6059d0c29b4b9d0220c7 not found: ID does not exist"
Nov 02 15:36:01.369960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.369920    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"]
Nov 02 15:36:01.382545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.382512    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-tvp64"]
Nov 02 15:36:01.867957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:01.867912    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00052150-74c8-4c65-8395-6b888102714b" path="/var/lib/kubelet/pods/00052150-74c8-4c65-8395-6b888102714b/volumes"
Nov 02 15:36:02.395170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:02.395135    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:02.579770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:02.579737    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:03.040838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.040802    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"]
Nov 02 15:36:03.052760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.052725    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a62e776e-8fjcn"]
Nov 02 15:36:03.390091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.390053    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68" podUID="08f14532-73ef-4bbd-9d31-2e2d0700f0b6" containerName="main" containerID="cri-o://235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf" gracePeriod=45
Nov 02 15:36:03.394864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.394821    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68" podUID="08f14532-73ef-4bbd-9d31-2e2d0700f0b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.159:8000/ready\": EOF"
Nov 02 15:36:03.669173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.669150    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:36:03.732382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.732346    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-dshm\") pod \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") "
Nov 02 15:36:03.732592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.732395    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tls-certs\") pod \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") "
Nov 02 15:36:03.732773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.732748    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-model-cache\") pod \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") "
Nov 02 15:36:03.733035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.732987    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-model-cache" pod "08f14532-73ef-4bbd-9d31-2e2d0700f0b6" (UID: "08f14532-73ef-4bbd-9d31-2e2d0700f0b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.733137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.733111    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-kube-api-access-vcn4c\" (UniqueName: \"kubernetes.io/projected/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-kube-api-access-vcn4c\") pod \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") "
Nov 02 15:36:03.733223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.733152    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tmp-dir\") pod \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") "
Nov 02 15:36:03.733223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.733181    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-home\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-home\") pod \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\" (UID: \"08f14532-73ef-4bbd-9d31-2e2d0700f0b6\") "
Nov 02 15:36:03.734215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.733868    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tmp-dir" pod "08f14532-73ef-4bbd-9d31-2e2d0700f0b6" (UID: "08f14532-73ef-4bbd-9d31-2e2d0700f0b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.734215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.734055    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-home" pod "08f14532-73ef-4bbd-9d31-2e2d0700f0b6" (UID: "08f14532-73ef-4bbd-9d31-2e2d0700f0b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.734215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.734176    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.734215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.734197    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.747251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.747069    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-dshm" pod "08f14532-73ef-4bbd-9d31-2e2d0700f0b6" (UID: "08f14532-73ef-4bbd-9d31-2e2d0700f0b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:03.747251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.747174    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tls-certs" pod "08f14532-73ef-4bbd-9d31-2e2d0700f0b6" (UID: "08f14532-73ef-4bbd-9d31-2e2d0700f0b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:03.747996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.747970    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-kube-api-access-vcn4c" pod "08f14532-73ef-4bbd-9d31-2e2d0700f0b6" (UID: "08f14532-73ef-4bbd-9d31-2e2d0700f0b6"). InnerVolumeSpecName "kube-api-access-vcn4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:03.836015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.835981    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcn4c\" (UniqueName: \"kubernetes.io/projected/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-kube-api-access-vcn4c\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.836015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.836012    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.836246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.836025    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.836246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.836036    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08f14532-73ef-4bbd-9d31-2e2d0700f0b6-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:03.865963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:03.865920    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4c9dce5-617c-4086-b13c-cd178b6bff07" path="/var/lib/kubelet/pods/c4c9dce5-617c-4086-b13c-cd178b6bff07/volumes"
Nov 02 15:36:04.067726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.063738    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:04.330698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.330574    7019 generic.go:356] "Generic (PLEG): container finished" podID="08f14532-73ef-4bbd-9d31-2e2d0700f0b6" containerID="235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf" exitCode=0
Nov 02 15:36:04.330698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.330628    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"
Nov 02 15:36:04.330698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.330684    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68" event={"ID":"08f14532-73ef-4bbd-9d31-2e2d0700f0b6","Type":"ContainerDied","Data":"235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf"}
Nov 02 15:36:04.331007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.330738    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68" event={"ID":"08f14532-73ef-4bbd-9d31-2e2d0700f0b6","Type":"ContainerDied","Data":"5040806dd3c8bfa1e704160597129e24d3c9d00e821e002862008f76209ce135"}
Nov 02 15:36:04.331007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.330763    7019 scope.go:122] "RemoveContainer" containerID="235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf"
Nov 02 15:36:04.343102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.343077    7019 scope.go:122] "RemoveContainer" containerID="235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf"
Nov 02 15:36:04.343440 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:36:04.343412    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf\": container with ID starting with 235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf not found: ID does not exist" containerID="235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf"
Nov 02 15:36:04.343538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.343451    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf"} err="failed to get container status \"235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf\": rpc error: code = NotFound desc = could not find container \"235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf\": container with ID starting with 235f2cc7cc06b66d1ad618661faa557f6ceeed0e489bf9c3b37251baf04719cf not found: ID does not exist"
Nov 02 15:36:04.412825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.412695    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"]
Nov 02 15:36:04.416752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:04.416721    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cfc63a/test-model-cfc63a-kserve-656bf444dc-x4t68"]
Nov 02 15:36:05.708673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:05.701923    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:05.864126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:05.864094    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f14532-73ef-4bbd-9d31-2e2d0700f0b6" path="/var/lib/kubelet/pods/08f14532-73ef-4bbd-9d31-2e2d0700f0b6/volumes"
Nov 02 15:36:07.317048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:07.317012    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:08.955256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:08.955208    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:10.254010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.253976    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"]
Nov 02 15:36:10.323423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.323372    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:10.793438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.793397    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:10.903998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.903963    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"]
Nov 02 15:36:10.904728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904705    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" containerName="revoke-keys"
Nov 02 15:36:10.904728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904725    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" containerName="revoke-keys"
Nov 02 15:36:10.904902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904748    7019 cpu_manager.go:414] "containerMap: removing container" podUID="bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" containerName="revoke-keys"
Nov 02 15:36:10.904902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904753    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" containerName="revoke-keys"
Nov 02 15:36:10.904902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904775    7019 cpu_manager.go:414] "containerMap: removing container" podUID="08f14532-73ef-4bbd-9d31-2e2d0700f0b6" containerName="main"
Nov 02 15:36:10.904902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904780    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f14532-73ef-4bbd-9d31-2e2d0700f0b6" containerName="main"
Nov 02 15:36:10.904902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904791    7019 cpu_manager.go:414] "containerMap: removing container" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy"
Nov 02 15:36:10.904902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904796    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy"
Nov 02 15:36:10.905101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904931    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" containerName="revoke-keys"
Nov 02 15:36:10.905101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904943    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" containerName="revoke-keys"
Nov 02 15:36:10.905101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904953    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f14532-73ef-4bbd-9d31-2e2d0700f0b6" containerName="main"
Nov 02 15:36:10.905101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.904961    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="00052150-74c8-4c65-8395-6b888102714b" containerName="istio-proxy"
Nov 02 15:36:10.910453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.910430    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:10.928863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.928836    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"]
Nov 02 15:36:10.937120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.937096    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:36:10.937447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.937425    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:36:10.979066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.979028    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsfk\" (UniqueName: \"kubernetes.io/projected/500fa23d-ea61-441a-9e7e-62dacd83331b-kube-api-access-hbsfk\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:10.979213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.979131    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:10.979332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:10.979315    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.080444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.080354    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsfk\" (UniqueName: \"kubernetes.io/projected/500fa23d-ea61-441a-9e7e-62dacd83331b-kube-api-access-hbsfk\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.080629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.080445    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.080760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.080685    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.081430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.081401    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.082961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.082942    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.104945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.104916    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsfk\" (UniqueName: \"kubernetes.io/projected/500fa23d-ea61-441a-9e7e-62dacd83331b-kube-api-access-hbsfk\") pod \"maas-api-revoke-sub-f407b067-hcw4t\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.227525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.227488    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:11.538147 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:36:11.538116    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500fa23d_ea61_441a_9e7e_62dacd83331b.slice/crio-172589ce77266a49e414bdde4fe7281978e89bf7a9c9b09c29eb1a54d77ef4eb WatchSource:0}: Error finding container 172589ce77266a49e414bdde4fe7281978e89bf7a9c9b09c29eb1a54d77ef4eb: Status 404 returned error can't find the container with id 172589ce77266a49e414bdde4fe7281978e89bf7a9c9b09c29eb1a54d77ef4eb
Nov 02 15:36:11.539868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:11.539844    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"]
Nov 02 15:36:12.411242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:12.410939    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:12.452133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:12.452098    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t" event={"ID":"500fa23d-ea61-441a-9e7e-62dacd83331b","Type":"ContainerStarted","Data":"05ff2f1c86957386246c0859b02c49b17fdbf1b70d0bf512acfa1eb65c8ce8e2"}
Nov 02 15:36:12.452295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:12.452152    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t" event={"ID":"500fa23d-ea61-441a-9e7e-62dacd83331b","Type":"ContainerStarted","Data":"172589ce77266a49e414bdde4fe7281978e89bf7a9c9b09c29eb1a54d77ef4eb"}
Nov 02 15:36:12.498465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:12.498411    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t" podStartSLOduration=2.498396945 podStartE2EDuration="2.498396945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:12.49455719 +0000 UTC m=+2366.748510816" watchObservedRunningTime="2026-11-02 15:36:12.498396945 +0000 UTC m=+2366.752350569"
Nov 02 15:36:13.383583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:13.383536    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"]
Nov 02 15:36:13.463416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:13.463374    7019 generic.go:356] "Generic (PLEG): container finished" podID="500fa23d-ea61-441a-9e7e-62dacd83331b" containerID="05ff2f1c86957386246c0859b02c49b17fdbf1b70d0bf512acfa1eb65c8ce8e2" exitCode=0
Nov 02 15:36:13.463595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:13.463456    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t" event={"ID":"500fa23d-ea61-441a-9e7e-62dacd83331b","Type":"ContainerDied","Data":"05ff2f1c86957386246c0859b02c49b17fdbf1b70d0bf512acfa1eb65c8ce8e2"}
Nov 02 15:36:13.506672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:13.506433    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:13.965638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:13.965597    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"]
Nov 02 15:36:13.970353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:13.970332    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:13.992325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:13.992290    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"]
Nov 02 15:36:14.041442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.041405    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.041679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.041635    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvx8j\" (UniqueName: \"kubernetes.io/projected/f031aa4e-da90-4dae-a7b4-e7124fb67d84-kube-api-access-vvx8j\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.041881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.041859    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.143198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.143159    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.143375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.143262    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.143467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.143417    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvx8j\" (UniqueName: \"kubernetes.io/projected/f031aa4e-da90-4dae-a7b4-e7124fb67d84-kube-api-access-vvx8j\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.143872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.143844    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.145712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.145679    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.170588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.170558    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvx8j\" (UniqueName: \"kubernetes.io/projected/f031aa4e-da90-4dae-a7b4-e7124fb67d84-kube-api-access-vvx8j\") pod \"maas-api-revoke-sub-c69bdd0d-6xr6f\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.226631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.226534    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:14.287493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.287449    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:14.612592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.609013    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"]
Nov 02 15:36:14.631117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.631095    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:14.752103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.752071    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-xks-ca\") pod \"500fa23d-ea61-441a-9e7e-62dacd83331b\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") "
Nov 02 15:36:14.752299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.752117    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500fa23d-ea61-441a-9e7e-62dacd83331b-kube-api-access-hbsfk\" (UniqueName: \"kubernetes.io/projected/500fa23d-ea61-441a-9e7e-62dacd83331b-kube-api-access-hbsfk\") pod \"500fa23d-ea61-441a-9e7e-62dacd83331b\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") "
Nov 02 15:36:14.752299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.752291    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-openshift-ca\") pod \"500fa23d-ea61-441a-9e7e-62dacd83331b\" (UID: \"500fa23d-ea61-441a-9e7e-62dacd83331b\") "
Nov 02 15:36:14.752796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.752765    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-openshift-ca" pod "500fa23d-ea61-441a-9e7e-62dacd83331b" (UID: "500fa23d-ea61-441a-9e7e-62dacd83331b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:14.752973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.752957    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:14.754165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.754138    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-xks-ca" pod "500fa23d-ea61-441a-9e7e-62dacd83331b" (UID: "500fa23d-ea61-441a-9e7e-62dacd83331b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:14.754291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.754239    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500fa23d-ea61-441a-9e7e-62dacd83331b-kube-api-access-hbsfk" pod "500fa23d-ea61-441a-9e7e-62dacd83331b" (UID: "500fa23d-ea61-441a-9e7e-62dacd83331b"). InnerVolumeSpecName "kube-api-access-hbsfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:14.854884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.854800    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fa23d-ea61-441a-9e7e-62dacd83331b-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:14.854884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:14.854828    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsfk\" (UniqueName: \"kubernetes.io/projected/500fa23d-ea61-441a-9e7e-62dacd83331b-kube-api-access-hbsfk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:15.056125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.056092    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"]
Nov 02 15:36:15.060162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.060133    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c00a77-r7gfx"]
Nov 02 15:36:15.482543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.482454    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"
Nov 02 15:36:15.482731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.482460    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t" event={"ID":"500fa23d-ea61-441a-9e7e-62dacd83331b","Type":"ContainerDied","Data":"172589ce77266a49e414bdde4fe7281978e89bf7a9c9b09c29eb1a54d77ef4eb"}
Nov 02 15:36:15.482731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.482591    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="172589ce77266a49e414bdde4fe7281978e89bf7a9c9b09c29eb1a54d77ef4eb"
Nov 02 15:36:15.484431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.484400    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f" event={"ID":"f031aa4e-da90-4dae-a7b4-e7124fb67d84","Type":"ContainerStarted","Data":"0542ccaf1d4b56e408572e6afdcf0c6f2ff629140400c1682f982b66028984bb"}
Nov 02 15:36:15.484571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.484443    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f" event={"ID":"f031aa4e-da90-4dae-a7b4-e7124fb67d84","Type":"ContainerStarted","Data":"d62264c20694db756841ca034fa193525d5b1ce70dec252c5dcc683cd62622c2"}
Nov 02 15:36:15.530898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.530850    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f" podStartSLOduration=2.530838232 podStartE2EDuration="2.530838232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:15.527528894 +0000 UTC m=+2369.781482517" watchObservedRunningTime="2026-11-02 15:36:15.530838232 +0000 UTC m=+2369.784791854"
Nov 02 15:36:15.863711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.863682    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5fe9cd-f10b-474b-b341-dcc7b0e5f087" path="/var/lib/kubelet/pods/7f5fe9cd-f10b-474b-b341-dcc7b0e5f087/volumes"
Nov 02 15:36:15.938973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:15.938937    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:16.497227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:16.497189    7019 generic.go:356] "Generic (PLEG): container finished" podID="f031aa4e-da90-4dae-a7b4-e7124fb67d84" containerID="0542ccaf1d4b56e408572e6afdcf0c6f2ff629140400c1682f982b66028984bb" exitCode=0
Nov 02 15:36:16.497441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:16.497275    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f" event={"ID":"f031aa4e-da90-4dae-a7b4-e7124fb67d84","Type":"ContainerDied","Data":"0542ccaf1d4b56e408572e6afdcf0c6f2ff629140400c1682f982b66028984bb"}
Nov 02 15:36:17.661700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.661369    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:17.766002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.765920    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:17.817709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.817675    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f031aa4e-da90-4dae-a7b4-e7124fb67d84-kube-api-access-vvx8j\" (UniqueName: \"kubernetes.io/projected/f031aa4e-da90-4dae-a7b4-e7124fb67d84-kube-api-access-vvx8j\") pod \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") "
Nov 02 15:36:17.817915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.817733    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-openshift-ca\") pod \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") "
Nov 02 15:36:17.817915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.817773    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-xks-ca\") pod \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\" (UID: \"f031aa4e-da90-4dae-a7b4-e7124fb67d84\") "
Nov 02 15:36:17.818120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.818090    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-openshift-ca" pod "f031aa4e-da90-4dae-a7b4-e7124fb67d84" (UID: "f031aa4e-da90-4dae-a7b4-e7124fb67d84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:17.818617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.818592    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:17.820066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.820042    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f031aa4e-da90-4dae-a7b4-e7124fb67d84-kube-api-access-vvx8j" pod "f031aa4e-da90-4dae-a7b4-e7124fb67d84" (UID: "f031aa4e-da90-4dae-a7b4-e7124fb67d84"). InnerVolumeSpecName "kube-api-access-vvx8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:17.820180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.820132    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-xks-ca" pod "f031aa4e-da90-4dae-a7b4-e7124fb67d84" (UID: "f031aa4e-da90-4dae-a7b4-e7124fb67d84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:17.920387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.920351    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvx8j\" (UniqueName: \"kubernetes.io/projected/f031aa4e-da90-4dae-a7b4-e7124fb67d84-kube-api-access-vvx8j\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:17.920387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:17.920389    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f031aa4e-da90-4dae-a7b4-e7124fb67d84-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:18.517961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:18.517924    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f" event={"ID":"f031aa4e-da90-4dae-a7b4-e7124fb67d84","Type":"ContainerDied","Data":"d62264c20694db756841ca034fa193525d5b1ce70dec252c5dcc683cd62622c2"}
Nov 02 15:36:18.517961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:18.517945    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"
Nov 02 15:36:18.518182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:18.517978    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d62264c20694db756841ca034fa193525d5b1ce70dec252c5dcc683cd62622c2"
Nov 02 15:36:19.279453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:19.279399    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:20.905269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:20.905209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:21.284205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.284120    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"]
Nov 02 15:36:21.284888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.284872    7019 cpu_manager.go:414] "containerMap: removing container" podUID="500fa23d-ea61-441a-9e7e-62dacd83331b" containerName="revoke-keys"
Nov 02 15:36:21.284931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.284889    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500fa23d-ea61-441a-9e7e-62dacd83331b" containerName="revoke-keys"
Nov 02 15:36:21.284931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.284918    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f031aa4e-da90-4dae-a7b4-e7124fb67d84" containerName="revoke-keys"
Nov 02 15:36:21.284931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.284924    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f031aa4e-da90-4dae-a7b4-e7124fb67d84" containerName="revoke-keys"
Nov 02 15:36:21.285042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.285033    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="500fa23d-ea61-441a-9e7e-62dacd83331b" containerName="revoke-keys"
Nov 02 15:36:21.285074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.285047    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f031aa4e-da90-4dae-a7b4-e7124fb67d84" containerName="revoke-keys"
Nov 02 15:36:21.289122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.289102    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.308610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.308579    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"]
Nov 02 15:36:21.315294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.315265    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:36:21.315535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.315514    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:36:21.376903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.376859    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.377095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.376967    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.377166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.377141    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdrc\" (UniqueName: \"kubernetes.io/projected/c989a692-390f-4ecf-a85b-bb0acf95adf9-kube-api-access-9rdrc\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.478533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.478491    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.478750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.478592    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.478750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.478631    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdrc\" (UniqueName: \"kubernetes.io/projected/c989a692-390f-4ecf-a85b-bb0acf95adf9-kube-api-access-9rdrc\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.479305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.479281    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.480933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.480907    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.505824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.505783    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdrc\" (UniqueName: \"kubernetes.io/projected/c989a692-390f-4ecf-a85b-bb0acf95adf9-kube-api-access-9rdrc\") pod \"maas-api-revoke-sub-e83dc05e-fcl76\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.605579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.605490    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:21.915558 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:36:21.915519    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc989a692_390f_4ecf_a85b_bb0acf95adf9.slice/crio-bac64f668fd0ab127d0c078be22b76409b103004559ece537343b7423221899c WatchSource:0}: Error finding container bac64f668fd0ab127d0c078be22b76409b103004559ece537343b7423221899c: Status 404 returned error can't find the container with id bac64f668fd0ab127d0c078be22b76409b103004559ece537343b7423221899c
Nov 02 15:36:21.916015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:21.915936    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"]
Nov 02 15:36:22.390295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:22.390170    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:22.601730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:22.597562    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76" event={"ID":"c989a692-390f-4ecf-a85b-bb0acf95adf9","Type":"ContainerStarted","Data":"0d01551d7ad7e3845e6eee926719ca964d64053450942b89542bbc6cd8f64344"}
Nov 02 15:36:22.601730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:22.597633    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76" event={"ID":"c989a692-390f-4ecf-a85b-bb0acf95adf9","Type":"ContainerStarted","Data":"bac64f668fd0ab127d0c078be22b76409b103004559ece537343b7423221899c"}
Nov 02 15:36:22.657843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:22.657517    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76" podStartSLOduration=1.657493007 podStartE2EDuration="1.657493007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:22.65209235 +0000 UTC m=+2376.906045975" watchObservedRunningTime="2026-11-02 15:36:22.657493007 +0000 UTC m=+2376.911446632"
Nov 02 15:36:22.909021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:22.903537    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:22.939109 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:36:22.939070    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:36:23.141306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:23.141270    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"]
Nov 02 15:36:23.147495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:23.147457    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-28f621-8gp5s"]
Nov 02 15:36:23.152605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:23.152578    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:23.610579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:23.610542    7019 generic.go:356] "Generic (PLEG): container finished" podID="c989a692-390f-4ecf-a85b-bb0acf95adf9" containerID="0d01551d7ad7e3845e6eee926719ca964d64053450942b89542bbc6cd8f64344" exitCode=0
Nov 02 15:36:23.610799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:23.610620    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76" event={"ID":"c989a692-390f-4ecf-a85b-bb0acf95adf9","Type":"ContainerDied","Data":"0d01551d7ad7e3845e6eee926719ca964d64053450942b89542bbc6cd8f64344"}
Nov 02 15:36:23.670954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:23.670919    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:23.863945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:23.863914    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c707aa-4f71-4f35-bfc7-0fc945e49a7f" path="/var/lib/kubelet/pods/d0c707aa-4f71-4f35-bfc7-0fc945e49a7f/volumes"
Nov 02 15:36:24.793916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.793888    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:24.860667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.860604    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c989a692-390f-4ecf-a85b-bb0acf95adf9-kube-api-access-9rdrc\" (UniqueName: \"kubernetes.io/projected/c989a692-390f-4ecf-a85b-bb0acf95adf9-kube-api-access-9rdrc\") pod \"c989a692-390f-4ecf-a85b-bb0acf95adf9\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") "
Nov 02 15:36:24.860864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.860710    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-openshift-ca\") pod \"c989a692-390f-4ecf-a85b-bb0acf95adf9\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") "
Nov 02 15:36:24.860995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.860972    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-xks-ca\") pod \"c989a692-390f-4ecf-a85b-bb0acf95adf9\" (UID: \"c989a692-390f-4ecf-a85b-bb0acf95adf9\") "
Nov 02 15:36:24.861372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.861347    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-openshift-ca" pod "c989a692-390f-4ecf-a85b-bb0acf95adf9" (UID: "c989a692-390f-4ecf-a85b-bb0acf95adf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:24.861669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.861611    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:24.863585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.863554    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c989a692-390f-4ecf-a85b-bb0acf95adf9-kube-api-access-9rdrc" pod "c989a692-390f-4ecf-a85b-bb0acf95adf9" (UID: "c989a692-390f-4ecf-a85b-bb0acf95adf9"). InnerVolumeSpecName "kube-api-access-9rdrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:24.863722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.863591    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-xks-ca" pod "c989a692-390f-4ecf-a85b-bb0acf95adf9" (UID: "c989a692-390f-4ecf-a85b-bb0acf95adf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:24.943744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.942303    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:24.962827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.962797    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989a692-390f-4ecf-a85b-bb0acf95adf9-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:24.962827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:24.962821    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdrc\" (UniqueName: \"kubernetes.io/projected/c989a692-390f-4ecf-a85b-bb0acf95adf9-kube-api-access-9rdrc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:25.277110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.277022    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t" podUID="9888de5c-77fa-4a4a-b931-11e46a4946a6" containerName="main" containerID="cri-o://d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d" gracePeriod=45
Nov 02 15:36:25.545201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.545175    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:36:25.575772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.575742    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-home\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-home\") pod \"9888de5c-77fa-4a4a-b931-11e46a4946a6\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") "
Nov 02 15:36:25.575955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.575843    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-dshm\") pod \"9888de5c-77fa-4a4a-b931-11e46a4946a6\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") "
Nov 02 15:36:25.575955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.575932    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-model-cache\") pod \"9888de5c-77fa-4a4a-b931-11e46a4946a6\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") "
Nov 02 15:36:25.576091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.575970    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-tmp-dir\") pod \"9888de5c-77fa-4a4a-b931-11e46a4946a6\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") "
Nov 02 15:36:25.576091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.576029    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9888de5c-77fa-4a4a-b931-11e46a4946a6-kube-api-access-9nb79\" (UniqueName: \"kubernetes.io/projected/9888de5c-77fa-4a4a-b931-11e46a4946a6-kube-api-access-9nb79\") pod \"9888de5c-77fa-4a4a-b931-11e46a4946a6\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") "
Nov 02 15:36:25.576091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.576045    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-home" pod "9888de5c-77fa-4a4a-b931-11e46a4946a6" (UID: "9888de5c-77fa-4a4a-b931-11e46a4946a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:25.576091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.576062    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9888de5c-77fa-4a4a-b931-11e46a4946a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/9888de5c-77fa-4a4a-b931-11e46a4946a6-tls-certs\") pod \"9888de5c-77fa-4a4a-b931-11e46a4946a6\" (UID: \"9888de5c-77fa-4a4a-b931-11e46a4946a6\") "
Nov 02 15:36:25.576312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.576225    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-model-cache" pod "9888de5c-77fa-4a4a-b931-11e46a4946a6" (UID: "9888de5c-77fa-4a4a-b931-11e46a4946a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:25.576571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.576545    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-tmp-dir" pod "9888de5c-77fa-4a4a-b931-11e46a4946a6" (UID: "9888de5c-77fa-4a4a-b931-11e46a4946a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:25.578111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.578091    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:25.578238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.578114    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:25.578238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.578128    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:25.578238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.578223    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-dshm" pod "9888de5c-77fa-4a4a-b931-11e46a4946a6" (UID: "9888de5c-77fa-4a4a-b931-11e46a4946a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:25.578613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.578590    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9888de5c-77fa-4a4a-b931-11e46a4946a6-kube-api-access-9nb79" pod "9888de5c-77fa-4a4a-b931-11e46a4946a6" (UID: "9888de5c-77fa-4a4a-b931-11e46a4946a6"). InnerVolumeSpecName "kube-api-access-9nb79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:25.578715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.578621    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9888de5c-77fa-4a4a-b931-11e46a4946a6-tls-certs" pod "9888de5c-77fa-4a4a-b931-11e46a4946a6" (UID: "9888de5c-77fa-4a4a-b931-11e46a4946a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:25.631587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.631539    7019 generic.go:356] "Generic (PLEG): container finished" podID="9888de5c-77fa-4a4a-b931-11e46a4946a6" containerID="d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d" exitCode=0
Nov 02 15:36:25.631823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.631594    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"
Nov 02 15:36:25.631823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.631621    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t" event={"ID":"9888de5c-77fa-4a4a-b931-11e46a4946a6","Type":"ContainerDied","Data":"d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d"}
Nov 02 15:36:25.631823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.631682    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t" event={"ID":"9888de5c-77fa-4a4a-b931-11e46a4946a6","Type":"ContainerDied","Data":"9ace56f3618243239248ab2f102bfc3b61bc2f37d477e61a1af4292a2c1e1cad"}
Nov 02 15:36:25.631823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.631707    7019 scope.go:122] "RemoveContainer" containerID="d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d"
Nov 02 15:36:25.633850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.633820    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76" event={"ID":"c989a692-390f-4ecf-a85b-bb0acf95adf9","Type":"ContainerDied","Data":"bac64f668fd0ab127d0c078be22b76409b103004559ece537343b7423221899c"}
Nov 02 15:36:25.633970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.633869    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bac64f668fd0ab127d0c078be22b76409b103004559ece537343b7423221899c"
Nov 02 15:36:25.633970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.633828    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"
Nov 02 15:36:25.645484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.645466    7019 scope.go:122] "RemoveContainer" containerID="d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d"
Nov 02 15:36:25.645828 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:36:25.645807    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d\": container with ID starting with d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d not found: ID does not exist" containerID="d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d"
Nov 02 15:36:25.645914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.645836    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d"} err="failed to get container status \"d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d\": rpc error: code = NotFound desc = could not find container \"d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d\": container with ID starting with d4631d0edd5c6147947fbf0c51e87a0e89b1836888e9c260aee766e909fe739d not found: ID does not exist"
Nov 02 15:36:25.680023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.679993    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nb79\" (UniqueName: \"kubernetes.io/projected/9888de5c-77fa-4a4a-b931-11e46a4946a6-kube-api-access-9nb79\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:25.680023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.680023    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9888de5c-77fa-4a4a-b931-11e46a4946a6-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:25.680195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.680035    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9888de5c-77fa-4a4a-b931-11e46a4946a6-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:25.689077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.689050    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"]
Nov 02 15:36:25.699822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.699794    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-80ec03/test-model-80ec03-kserve-994f47f6f-xrk7t"]
Nov 02 15:36:25.865363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:25.865331    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9888de5c-77fa-4a4a-b931-11e46a4946a6" path="/var/lib/kubelet/pods/9888de5c-77fa-4a4a-b931-11e46a4946a6/volumes"
Nov 02 15:36:26.488723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:26.488686    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:28.177670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.177532    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:28.420710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.420625    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j" podUID="b4438617-bb51-496d-aee4-870ca57e9907" containerName="main" containerID="cri-o://bb178cd1e88a9ee045a71b41d7b001b0829828951addf4ab48889a5195c4cd17" gracePeriod=45
Nov 02 15:36:28.662556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.662512    7019 generic.go:356] "Generic (PLEG): container finished" podID="b4438617-bb51-496d-aee4-870ca57e9907" containerID="bb178cd1e88a9ee045a71b41d7b001b0829828951addf4ab48889a5195c4cd17" exitCode=0
Nov 02 15:36:28.662787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.662594    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j" event={"ID":"b4438617-bb51-496d-aee4-870ca57e9907","Type":"ContainerDied","Data":"bb178cd1e88a9ee045a71b41d7b001b0829828951addf4ab48889a5195c4cd17"}
Nov 02 15:36:28.734843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.734816    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:36:28.832304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832261    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-model-cache\") pod \"b4438617-bb51-496d-aee4-870ca57e9907\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") "
Nov 02 15:36:28.832477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832337    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-home\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-home\") pod \"b4438617-bb51-496d-aee4-870ca57e9907\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") "
Nov 02 15:36:28.832477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832404    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-tmp-dir\") pod \"b4438617-bb51-496d-aee4-870ca57e9907\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") "
Nov 02 15:36:28.832557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832509    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4438617-bb51-496d-aee4-870ca57e9907-kube-api-access-j4lpd\" (UniqueName: \"kubernetes.io/projected/b4438617-bb51-496d-aee4-870ca57e9907-kube-api-access-j4lpd\") pod \"b4438617-bb51-496d-aee4-870ca57e9907\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") "
Nov 02 15:36:28.832597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832559    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-model-cache" pod "b4438617-bb51-496d-aee4-870ca57e9907" (UID: "b4438617-bb51-496d-aee4-870ca57e9907"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:28.832597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832582    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-home" pod "b4438617-bb51-496d-aee4-870ca57e9907" (UID: "b4438617-bb51-496d-aee4-870ca57e9907"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:28.832730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832682    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4438617-bb51-496d-aee4-870ca57e9907-tls-certs\" (UniqueName: \"kubernetes.io/secret/b4438617-bb51-496d-aee4-870ca57e9907-tls-certs\") pod \"b4438617-bb51-496d-aee4-870ca57e9907\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") "
Nov 02 15:36:28.832730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832719    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-dshm\") pod \"b4438617-bb51-496d-aee4-870ca57e9907\" (UID: \"b4438617-bb51-496d-aee4-870ca57e9907\") "
Nov 02 15:36:28.833119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.832808    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-tmp-dir" pod "b4438617-bb51-496d-aee4-870ca57e9907" (UID: "b4438617-bb51-496d-aee4-870ca57e9907"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:28.833298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.833282    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:28.833388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.833303    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:28.833388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.833315    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:28.835069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.835043    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4438617-bb51-496d-aee4-870ca57e9907-tls-certs" pod "b4438617-bb51-496d-aee4-870ca57e9907" (UID: "b4438617-bb51-496d-aee4-870ca57e9907"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:28.835296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.835275    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-dshm" pod "b4438617-bb51-496d-aee4-870ca57e9907" (UID: "b4438617-bb51-496d-aee4-870ca57e9907"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:28.835350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.835303    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4438617-bb51-496d-aee4-870ca57e9907-kube-api-access-j4lpd" pod "b4438617-bb51-496d-aee4-870ca57e9907" (UID: "b4438617-bb51-496d-aee4-870ca57e9907"). InnerVolumeSpecName "kube-api-access-j4lpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:28.934439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.934404    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lpd\" (UniqueName: \"kubernetes.io/projected/b4438617-bb51-496d-aee4-870ca57e9907-kube-api-access-j4lpd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:28.934439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.934435    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4438617-bb51-496d-aee4-870ca57e9907-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:28.934439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:28.934445    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4438617-bb51-496d-aee4-870ca57e9907-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:29.673829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:29.673796    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"
Nov 02 15:36:29.674291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:29.673803    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j" event={"ID":"b4438617-bb51-496d-aee4-870ca57e9907","Type":"ContainerDied","Data":"ef19d8b90c8e9aff28afb98ff12b1dd76c47e1372f8cc708a93f8f720691e0a2"}
Nov 02 15:36:29.674291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:29.673945    7019 scope.go:122] "RemoveContainer" containerID="bb178cd1e88a9ee045a71b41d7b001b0829828951addf4ab48889a5195c4cd17"
Nov 02 15:36:29.729627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:29.729593    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"]
Nov 02 15:36:29.744174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:29.744142    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2f8b7c/test-model-2f8b7c-kserve-7c56cc97cf-96x4j"]
Nov 02 15:36:29.864298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:29.864267    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4438617-bb51-496d-aee4-870ca57e9907" path="/var/lib/kubelet/pods/b4438617-bb51-496d-aee4-870ca57e9907/volumes"
Nov 02 15:36:30.005538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:29.998597    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:30.671248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:30.671206    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:30.848073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:30.848034    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:31.006721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.005798    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"]
Nov 02 15:36:31.006721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.006027    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy" containerID="cri-o://65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f" gracePeriod=30
Nov 02 15:36:31.039375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.039340    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:36:31.070277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.070239    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:31.687108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.687073    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"]
Nov 02 15:36:31.688166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688139    7019 cpu_manager.go:414] "containerMap: removing container" podUID="9888de5c-77fa-4a4a-b931-11e46a4946a6" containerName="main"
Nov 02 15:36:31.688166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688163    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9888de5c-77fa-4a4a-b931-11e46a4946a6" containerName="main"
Nov 02 15:36:31.688367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688190    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b4438617-bb51-496d-aee4-870ca57e9907" containerName="main"
Nov 02 15:36:31.688367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688198    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4438617-bb51-496d-aee4-870ca57e9907" containerName="main"
Nov 02 15:36:31.688367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688269    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c989a692-390f-4ecf-a85b-bb0acf95adf9" containerName="revoke-keys"
Nov 02 15:36:31.688367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688277    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c989a692-390f-4ecf-a85b-bb0acf95adf9" containerName="revoke-keys"
Nov 02 15:36:31.688573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688435    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c989a692-390f-4ecf-a85b-bb0acf95adf9" containerName="revoke-keys"
Nov 02 15:36:31.688573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688453    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9888de5c-77fa-4a4a-b931-11e46a4946a6" containerName="main"
Nov 02 15:36:31.688573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.688464    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4438617-bb51-496d-aee4-870ca57e9907" containerName="main"
Nov 02 15:36:31.694277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.694253    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.718512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.715501    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:36:31.723479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.723174    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"]
Nov 02 15:36:31.728665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.726933    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:36:31.787816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.787780    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7cl\" (UniqueName: \"kubernetes.io/projected/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-kube-api-access-cg7cl\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.788004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.787980    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.788067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.788027    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.889227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.889193    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.889770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.889260    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.889770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.889343    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7cl\" (UniqueName: \"kubernetes.io/projected/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-kube-api-access-cg7cl\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.889877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.889855    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.891897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.891866    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:31.916923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:31.916887    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7cl\" (UniqueName: \"kubernetes.io/projected/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-kube-api-access-cg7cl\") pod \"maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:32.010044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:32.009950    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:32.014586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:32.014559    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:32.393414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:32.393371    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"]
Nov 02 15:36:32.758669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:32.758564    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb" event={"ID":"978d43d5-e5d0-4f6e-8727-aadb6fa066ab","Type":"ContainerStarted","Data":"af123df37e2a562215f9c6d656e28e82615d41bdaea9249c6333c12fc75cb5d6"}
Nov 02 15:36:33.768109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:33.768027    7019 generic.go:356] "Generic (PLEG): container finished" podID="978d43d5-e5d0-4f6e-8727-aadb6fa066ab" containerID="66915c0b04432e9074c42c961a2356a59bc901a5d57f8cb2ef63e9409c2dc7d9" exitCode=0
Nov 02 15:36:33.768481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:33.768114    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb" event={"ID":"978d43d5-e5d0-4f6e-8727-aadb6fa066ab","Type":"ContainerDied","Data":"66915c0b04432e9074c42c961a2356a59bc901a5d57f8cb2ef63e9409c2dc7d9"}
Nov 02 15:36:33.997014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:33.996977    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:34.949850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:34.949824    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:35.057364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.057267    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-openshift-ca\") pod \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") "
Nov 02 15:36:35.057364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.057347    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-xks-ca\") pod \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") "
Nov 02 15:36:35.057608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.057468    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-kube-api-access-cg7cl\" (UniqueName: \"kubernetes.io/projected/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-kube-api-access-cg7cl\") pod \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\" (UID: \"978d43d5-e5d0-4f6e-8727-aadb6fa066ab\") "
Nov 02 15:36:35.058246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.058201    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-openshift-ca" pod "978d43d5-e5d0-4f6e-8727-aadb6fa066ab" (UID: "978d43d5-e5d0-4f6e-8727-aadb6fa066ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:35.060965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.060876    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-kube-api-access-cg7cl" pod "978d43d5-e5d0-4f6e-8727-aadb6fa066ab" (UID: "978d43d5-e5d0-4f6e-8727-aadb6fa066ab"). InnerVolumeSpecName "kube-api-access-cg7cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:35.061239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.061214    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-xks-ca" pod "978d43d5-e5d0-4f6e-8727-aadb6fa066ab" (UID: "978d43d5-e5d0-4f6e-8727-aadb6fa066ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:35.159550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.159513    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:35.159550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.159543    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:35.159550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.159553    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7cl\" (UniqueName: \"kubernetes.io/projected/978d43d5-e5d0-4f6e-8727-aadb6fa066ab-kube-api-access-cg7cl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:35.694571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.694529    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:35.787524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.787483    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"
Nov 02 15:36:35.787524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.787520    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb" event={"ID":"978d43d5-e5d0-4f6e-8727-aadb6fa066ab","Type":"ContainerDied","Data":"af123df37e2a562215f9c6d656e28e82615d41bdaea9249c6333c12fc75cb5d6"}
Nov 02 15:36:35.787799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:35.787568    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af123df37e2a562215f9c6d656e28e82615d41bdaea9249c6333c12fc75cb5d6"
Nov 02 15:36:36.322378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.322351    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:36:36.483128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483080    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-podinfo\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483206    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-socket\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483277    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-envoy\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483353    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-data\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483436    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-certs\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483507    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-kube-api-access-fmddn\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-kube-api-access-fmddn\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483516    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-socket" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.483602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483564    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-credential-socket\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483625    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-data" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.483800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483723    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b46866-6bf6-4935-ae33-23d0c1579263-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b46866-6bf6-4935-ae33-23d0c1579263-istiod-ca-cert\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.483800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483748    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-certs" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.483800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.483785    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-token\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-token\") pod \"b4b46866-6bf6-4935-ae33-23d0c1579263\" (UID: \"b4b46866-6bf6-4935-ae33-23d0c1579263\") "
Nov 02 15:36:36.484056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.484033    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-credential-socket" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.484520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.484492    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.484628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.484527    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.484628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.484545    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.484628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.484558    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.484836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.484786    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b46866-6bf6-4935-ae33-23d0c1579263-istiod-ca-cert" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:36.486171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.486142    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-envoy" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:36.486293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.486191    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-kube-api-access-fmddn" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "kube-api-access-fmddn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:36.486293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.486256    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-podinfo" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:36:36.486412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.486297    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-token" pod "b4b46866-6bf6-4935-ae33-23d0c1579263" (UID: "b4b46866-6bf6-4935-ae33-23d0c1579263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:36.586466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.586364    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4b46866-6bf6-4935-ae33-23d0c1579263-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.586466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.586404    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.586466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.586417    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.586466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.586429    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4b46866-6bf6-4935-ae33-23d0c1579263-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.586466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.586442    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmddn\" (UniqueName: \"kubernetes.io/projected/b4b46866-6bf6-4935-ae33-23d0c1579263-kube-api-access-fmddn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:36.798729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.798686    7019 generic.go:356] "Generic (PLEG): container finished" podID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerID="65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f" exitCode=0
Nov 02 15:36:36.798914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.798738    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" event={"ID":"b4b46866-6bf6-4935-ae33-23d0c1579263","Type":"ContainerDied","Data":"65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f"}
Nov 02 15:36:36.798914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.798801    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"
Nov 02 15:36:36.798914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.798819    7019 scope.go:122] "RemoveContainer" containerID="65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f"
Nov 02 15:36:36.798914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.798807    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p" event={"ID":"b4b46866-6bf6-4935-ae33-23d0c1579263","Type":"ContainerDied","Data":"e64687436a27cb1753b9aea06bb6fa83d1a2b27d35dd08d5738186fec8547ab1"}
Nov 02 15:36:36.811316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.811295    7019 scope.go:122] "RemoveContainer" containerID="65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f"
Nov 02 15:36:36.811598 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:36:36.811577    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f\": container with ID starting with 65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f not found: ID does not exist" containerID="65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f"
Nov 02 15:36:36.811707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.811606    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f"} err="failed to get container status \"65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f\": rpc error: code = NotFound desc = could not find container \"65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f\": container with ID starting with 65802858dce96dbe04b34406ba9edd66735417a8647a5f51e431ac60dd00d75f not found: ID does not exist"
Nov 02 15:36:36.855611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.855531    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"]
Nov 02 15:36:36.868427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:36.868391    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-28f621-openshift-default-7dd8c5c789-wvg2p"]
Nov 02 15:36:37.366084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:37.366046    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:37.863136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:37.863105    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" path="/var/lib/kubelet/pods/b4b46866-6bf6-4935-ae33-23d0c1579263/volumes"
Nov 02 15:36:39.279830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:39.279794    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:40.037556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:40.037513    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"]
Nov 02 15:36:40.050150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:40.050116    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859260fd-545p8"]
Nov 02 15:36:40.850611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:40.850575    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:41.804832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:41.804791    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"]
Nov 02 15:36:41.856510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:41.856477    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:41.864393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:41.864360    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13dd0439-d5a2-451f-ab15-cf39bca9c51f" path="/var/lib/kubelet/pods/13dd0439-d5a2-451f-ab15-cf39bca9c51f/volumes"
Nov 02 15:36:42.074776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.074678    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"]
Nov 02 15:36:42.075550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.075519    7019 cpu_manager.go:414] "containerMap: removing container" podUID="978d43d5-e5d0-4f6e-8727-aadb6fa066ab" containerName="revoke-keys"
Nov 02 15:36:42.075550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.075543    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978d43d5-e5d0-4f6e-8727-aadb6fa066ab" containerName="revoke-keys"
Nov 02 15:36:42.075811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.075566    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy"
Nov 02 15:36:42.075811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.075572    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy"
Nov 02 15:36:42.075811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.075727    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="978d43d5-e5d0-4f6e-8727-aadb6fa066ab" containerName="revoke-keys"
Nov 02 15:36:42.075811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.075742    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b46866-6bf6-4935-ae33-23d0c1579263" containerName="istio-proxy"
Nov 02 15:36:42.079733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.079714    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.100090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.100060    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"]
Nov 02 15:36:42.100920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.100877    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:36:42.101157 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.101138    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:36:42.211921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.211873    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da823233-c848-48c6-8855-36c6787b174d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.212223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.212193    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da823233-c848-48c6-8855-36c6787b174d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.212346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.212324    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqb6\" (UniqueName: \"kubernetes.io/projected/da823233-c848-48c6-8855-36c6787b174d-kube-api-access-gsqb6\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.313604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.313572    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da823233-c848-48c6-8855-36c6787b174d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.313863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.313804    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da823233-c848-48c6-8855-36c6787b174d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.313933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.313873    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqb6\" (UniqueName: \"kubernetes.io/projected/da823233-c848-48c6-8855-36c6787b174d-kube-api-access-gsqb6\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.314474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.314443    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da823233-c848-48c6-8855-36c6787b174d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.316251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.316221    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da823233-c848-48c6-8855-36c6787b174d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.338793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.338735    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqb6\" (UniqueName: \"kubernetes.io/projected/da823233-c848-48c6-8855-36c6787b174d-kube-api-access-gsqb6\") pod \"maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.399409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.399376    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:42.729550 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:36:42.729513    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda823233_c848_48c6_8855_36c6787b174d.slice/crio-3cb64c1b206915c09e394d3c397c674ee95d6b84eb64fe2efe9b1b26d30a75e9 WatchSource:0}: Error finding container 3cb64c1b206915c09e394d3c397c674ee95d6b84eb64fe2efe9b1b26d30a75e9: Status 404 returned error can't find the container with id 3cb64c1b206915c09e394d3c397c674ee95d6b84eb64fe2efe9b1b26d30a75e9
Nov 02 15:36:42.729737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.729717    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"]
Nov 02 15:36:42.744007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.743976    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:42.852599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:42.852563    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9" event={"ID":"da823233-c848-48c6-8855-36c6787b174d","Type":"ContainerStarted","Data":"3cb64c1b206915c09e394d3c397c674ee95d6b84eb64fe2efe9b1b26d30a75e9"}
Nov 02 15:36:43.089295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:43.089260    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:43.235239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:43.235205    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:43.864329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:43.864297    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9" event={"ID":"da823233-c848-48c6-8855-36c6787b174d","Type":"ContainerStarted","Data":"0a9cf682be23e64ddebbebfb70f6aca2d40bb2e2a5a56a65f4b0f6296622fb81"}
Nov 02 15:36:44.606892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:44.606832    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:44.915693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:44.896148    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9" podStartSLOduration=2.896127982 podStartE2EDuration="2.896127982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:43.909170686 +0000 UTC m=+2398.163124309" watchObservedRunningTime="2026-11-02 15:36:44.896127982 +0000 UTC m=+2399.150081608"
Nov 02 15:36:44.924417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:44.924379    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"]
Nov 02 15:36:44.925811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:44.925733    7019 generic.go:356] "Generic (PLEG): container finished" podID="da823233-c848-48c6-8855-36c6787b174d" containerID="0a9cf682be23e64ddebbebfb70f6aca2d40bb2e2a5a56a65f4b0f6296622fb81" exitCode=0
Nov 02 15:36:44.926502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:44.926469    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9" event={"ID":"da823233-c848-48c6-8855-36c6787b174d","Type":"ContainerDied","Data":"0a9cf682be23e64ddebbebfb70f6aca2d40bb2e2a5a56a65f4b0f6296622fb81"}
Nov 02 15:36:44.978428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:44.978261    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:45.355757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.355636    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"]
Nov 02 15:36:45.496674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.474081    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:45.519671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.507707    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"]
Nov 02 15:36:45.519671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.513814    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.556183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.556134    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"]
Nov 02 15:36:45.621215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.621174    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cp2l\" (UniqueName: \"kubernetes.io/projected/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-kube-api-access-9cp2l\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.621609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.621241    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.621609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.621497    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.633376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.633340    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"]
Nov 02 15:36:45.638143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.638118    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.658260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.658228    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"]
Nov 02 15:36:45.722732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.722700    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.722915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.722818    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cp2l\" (UniqueName: \"kubernetes.io/projected/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-kube-api-access-9cp2l\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.722915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.722853    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.723778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.723751    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.725184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.725161    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.747167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.747137    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cp2l\" (UniqueName: \"kubernetes.io/projected/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-kube-api-access-9cp2l\") pod \"maas-api-revoke-sub-9b3e6873-vsl79\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.824953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.824910    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.825134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.824981    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qb9d\" (UniqueName: \"kubernetes.io/projected/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-kube-api-access-4qb9d\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.825185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.825164    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.849685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.849638    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:45.930670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.927077    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.930670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.927238    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.930670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.927269    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qb9d\" (UniqueName: \"kubernetes.io/projected/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-kube-api-access-4qb9d\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.930670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.927941    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.940797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.940586    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.954356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.954310    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qb9d\" (UniqueName: \"kubernetes.io/projected/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-kube-api-access-4qb9d\") pod \"maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:45.954594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:45.954577    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:46.147744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.147714    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:46.188557 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:36:46.188521    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7dc0f1_0a16_41ab_99cb_478903dddeb2.slice/crio-f8af5cdc0a700c461090409f3ef339cc77bf5f33464cea496ecc7518b0e45d55 WatchSource:0}: Error finding container f8af5cdc0a700c461090409f3ef339cc77bf5f33464cea496ecc7518b0e45d55: Status 404 returned error can't find the container with id f8af5cdc0a700c461090409f3ef339cc77bf5f33464cea496ecc7518b0e45d55
Nov 02 15:36:46.188898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.188874    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"]
Nov 02 15:36:46.320863 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:36:46.320833    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a81c2bf_82f1_448f_8bd8_df6bc7f9b7b1.slice/crio-eed7fd963cb9863aadddb823f38b0b3c3db7bd9c98b4031b32c61b2aab542d50 WatchSource:0}: Error finding container eed7fd963cb9863aadddb823f38b0b3c3db7bd9c98b4031b32c61b2aab542d50: Status 404 returned error can't find the container with id eed7fd963cb9863aadddb823f38b0b3c3db7bd9c98b4031b32c61b2aab542d50
Nov 02 15:36:46.322228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.322202    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"]
Nov 02 15:36:46.332682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.332617    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da823233-c848-48c6-8855-36c6787b174d-kube-api-access-gsqb6\" (UniqueName: \"kubernetes.io/projected/da823233-c848-48c6-8855-36c6787b174d-kube-api-access-gsqb6\") pod \"da823233-c848-48c6-8855-36c6787b174d\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") "
Nov 02 15:36:46.332832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.332813    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da823233-c848-48c6-8855-36c6787b174d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da823233-c848-48c6-8855-36c6787b174d-maas-api-xks-ca\") pod \"da823233-c848-48c6-8855-36c6787b174d\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") "
Nov 02 15:36:46.332986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.332966    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da823233-c848-48c6-8855-36c6787b174d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da823233-c848-48c6-8855-36c6787b174d-maas-api-openshift-ca\") pod \"da823233-c848-48c6-8855-36c6787b174d\" (UID: \"da823233-c848-48c6-8855-36c6787b174d\") "
Nov 02 15:36:46.333585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.333562    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da823233-c848-48c6-8855-36c6787b174d-maas-api-openshift-ca" pod "da823233-c848-48c6-8855-36c6787b174d" (UID: "da823233-c848-48c6-8855-36c6787b174d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:46.333839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.333815    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da823233-c848-48c6-8855-36c6787b174d-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:46.335135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.335112    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da823233-c848-48c6-8855-36c6787b174d-kube-api-access-gsqb6" pod "da823233-c848-48c6-8855-36c6787b174d" (UID: "da823233-c848-48c6-8855-36c6787b174d"). InnerVolumeSpecName "kube-api-access-gsqb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:46.335353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.335325    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da823233-c848-48c6-8855-36c6787b174d-maas-api-xks-ca" pod "da823233-c848-48c6-8855-36c6787b174d" (UID: "da823233-c848-48c6-8855-36c6787b174d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:46.420319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.420227    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:46.439788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.439714    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqb6\" (UniqueName: \"kubernetes.io/projected/da823233-c848-48c6-8855-36c6787b174d-kube-api-access-gsqb6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:46.439788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.439756    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da823233-c848-48c6-8855-36c6787b174d-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:46.961212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.961125    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79" event={"ID":"7a7dc0f1-0a16-41ab-99cb-478903dddeb2","Type":"ContainerStarted","Data":"568e8c55f2f906bd95b2ebe13913faf5016ed743d857763cd54990f877ec5d6e"}
Nov 02 15:36:46.961592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.961222    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79" event={"ID":"7a7dc0f1-0a16-41ab-99cb-478903dddeb2","Type":"ContainerStarted","Data":"f8af5cdc0a700c461090409f3ef339cc77bf5f33464cea496ecc7518b0e45d55"}
Nov 02 15:36:46.962850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.962812    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9" event={"ID":"da823233-c848-48c6-8855-36c6787b174d","Type":"ContainerDied","Data":"3cb64c1b206915c09e394d3c397c674ee95d6b84eb64fe2efe9b1b26d30a75e9"}
Nov 02 15:36:46.962976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.962866    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"
Nov 02 15:36:46.962976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.962885    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb64c1b206915c09e394d3c397c674ee95d6b84eb64fe2efe9b1b26d30a75e9"
Nov 02 15:36:46.964487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.964466    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8" event={"ID":"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1","Type":"ContainerStarted","Data":"eb63fc3bde74defc746201596ac3ab16cf71c4cb281c0a52a7cd1a4ca741aac0"}
Nov 02 15:36:46.964572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:46.964501    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8" event={"ID":"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1","Type":"ContainerStarted","Data":"eed7fd963cb9863aadddb823f38b0b3c3db7bd9c98b4031b32c61b2aab542d50"}
Nov 02 15:36:47.006946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:47.006896    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79" podStartSLOduration=2.006881408 podStartE2EDuration="2.006881408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:47.004891099 +0000 UTC m=+2401.258844724" watchObservedRunningTime="2026-11-02 15:36:47.006881408 +0000 UTC m=+2401.260835025"
Nov 02 15:36:47.053623 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:47.053571    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8" podStartSLOduration=2.053556722 podStartE2EDuration="2.053556722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:47.052006415 +0000 UTC m=+2401.305960051" watchObservedRunningTime="2026-11-02 15:36:47.053556722 +0000 UTC m=+2401.307510345"
Nov 02 15:36:47.975630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:47.975584    7019 generic.go:356] "Generic (PLEG): container finished" podID="1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" containerID="eb63fc3bde74defc746201596ac3ab16cf71c4cb281c0a52a7cd1a4ca741aac0" exitCode=0
Nov 02 15:36:47.976152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:47.975677    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8" event={"ID":"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1","Type":"ContainerDied","Data":"eb63fc3bde74defc746201596ac3ab16cf71c4cb281c0a52a7cd1a4ca741aac0"}
Nov 02 15:36:47.977610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:47.977585    7019 generic.go:356] "Generic (PLEG): container finished" podID="7a7dc0f1-0a16-41ab-99cb-478903dddeb2" containerID="568e8c55f2f906bd95b2ebe13913faf5016ed743d857763cd54990f877ec5d6e" exitCode=0
Nov 02 15:36:47.977751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:47.977678    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79" event={"ID":"7a7dc0f1-0a16-41ab-99cb-478903dddeb2","Type":"ContainerDied","Data":"568e8c55f2f906bd95b2ebe13913faf5016ed743d857763cd54990f877ec5d6e"}
Nov 02 15:36:48.196576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:48.196537    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:49.046488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.046458    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:49.205958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.203953    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:49.252489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.252455    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:49.292917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.292881    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-xks-ca\") pod \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") "
Nov 02 15:36:49.293124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.292941    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-openshift-ca\") pod \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") "
Nov 02 15:36:49.293124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.292987    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-xks-ca\") pod \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") "
Nov 02 15:36:49.293256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.293156    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-kube-api-access-9cp2l\" (UniqueName: \"kubernetes.io/projected/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-kube-api-access-9cp2l\") pod \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") "
Nov 02 15:36:49.293256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.293182    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-kube-api-access-4qb9d\" (UniqueName: \"kubernetes.io/projected/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-kube-api-access-4qb9d\") pod \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\" (UID: \"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1\") "
Nov 02 15:36:49.293391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.293362    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-openshift-ca\") pod \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\" (UID: \"7a7dc0f1-0a16-41ab-99cb-478903dddeb2\") "
Nov 02 15:36:49.294564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.294463    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-openshift-ca" pod "7a7dc0f1-0a16-41ab-99cb-478903dddeb2" (UID: "7a7dc0f1-0a16-41ab-99cb-478903dddeb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:49.294564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.294527    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-openshift-ca" pod "1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" (UID: "1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:36:49.324085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.322542    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-xks-ca" pod "1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" (UID: "1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:49.324085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.323896    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-kube-api-access-9cp2l" pod "7a7dc0f1-0a16-41ab-99cb-478903dddeb2" (UID: "7a7dc0f1-0a16-41ab-99cb-478903dddeb2"). InnerVolumeSpecName "kube-api-access-9cp2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:49.324085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.324017    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-xks-ca" pod "7a7dc0f1-0a16-41ab-99cb-478903dddeb2" (UID: "7a7dc0f1-0a16-41ab-99cb-478903dddeb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:49.324085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.324068    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-kube-api-access-4qb9d" pod "1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" (UID: "1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1"). InnerVolumeSpecName "kube-api-access-4qb9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:49.396177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.396138    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:49.396177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.396180    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cp2l\" (UniqueName: \"kubernetes.io/projected/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-kube-api-access-9cp2l\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:49.396387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.396199    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qb9d\" (UniqueName: \"kubernetes.io/projected/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-kube-api-access-4qb9d\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:49.396387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.396214    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7dc0f1-0a16-41ab-99cb-478903dddeb2-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:49.396387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.396230    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:49.396387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.396243    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:49.908895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:49.900837    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:50.001784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:50.001750    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"
Nov 02 15:36:50.001784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:50.001762    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79" event={"ID":"7a7dc0f1-0a16-41ab-99cb-478903dddeb2","Type":"ContainerDied","Data":"f8af5cdc0a700c461090409f3ef339cc77bf5f33464cea496ecc7518b0e45d55"}
Nov 02 15:36:50.002048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:50.001826    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8af5cdc0a700c461090409f3ef339cc77bf5f33464cea496ecc7518b0e45d55"
Nov 02 15:36:50.003599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:50.003562    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8" event={"ID":"1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1","Type":"ContainerDied","Data":"eed7fd963cb9863aadddb823f38b0b3c3db7bd9c98b4031b32c61b2aab542d50"}
Nov 02 15:36:50.003599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:50.003593    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"
Nov 02 15:36:50.003841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:50.003609    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eed7fd963cb9863aadddb823f38b0b3c3db7bd9c98b4031b32c61b2aab542d50"
Nov 02 15:36:51.478708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:51.477752    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:52.722385 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:36:52.722345    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:36:52.737799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:52.737762    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:52.884903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:52.884870    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:53.033834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:53.033741    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:54.602961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:54.602886    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:55.982473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:55.982435    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:56.146757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.146718    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"]
Nov 02 15:36:56.360346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.360249    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:56.527392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.527354    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:56.673086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.673046    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"]
Nov 02 15:36:56.680887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.680851    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2f8b7c-kn6j9"]
Nov 02 15:36:56.712938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.712896    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:56.838889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.838845    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf" podUID="2e158e74-74d1-473c-a673-71119444b264" containerName="main" containerID="cri-o://9a78dfbf474adc70dae8a8e167524c8fe6fc7755ba59b9e092871032c3d7f395" gracePeriod=45
Nov 02 15:36:56.874891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:56.874857    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:57.087179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.087143    7019 generic.go:356] "Generic (PLEG): container finished" podID="2e158e74-74d1-473c-a673-71119444b264" containerID="9a78dfbf474adc70dae8a8e167524c8fe6fc7755ba59b9e092871032c3d7f395" exitCode=0
Nov 02 15:36:57.087947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.087920    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf" event={"ID":"2e158e74-74d1-473c-a673-71119444b264","Type":"ContainerDied","Data":"9a78dfbf474adc70dae8a8e167524c8fe6fc7755ba59b9e092871032c3d7f395"}
Nov 02 15:36:57.165928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.165896    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:36:57.232437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.232353    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:57.248208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.248169    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-home\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-home\") pod \"2e158e74-74d1-473c-a673-71119444b264\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") "
Nov 02 15:36:57.248495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.248453    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-home" pod "2e158e74-74d1-473c-a673-71119444b264" (UID: "2e158e74-74d1-473c-a673-71119444b264"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:57.248495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.248460    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-dshm\") pod \"2e158e74-74d1-473c-a673-71119444b264\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") "
Nov 02 15:36:57.248755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.248541    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-tmp-dir\") pod \"2e158e74-74d1-473c-a673-71119444b264\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") "
Nov 02 15:36:57.248755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.248593    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e158e74-74d1-473c-a673-71119444b264-kube-api-access-xt7fx\" (UniqueName: \"kubernetes.io/projected/2e158e74-74d1-473c-a673-71119444b264-kube-api-access-xt7fx\") pod \"2e158e74-74d1-473c-a673-71119444b264\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") "
Nov 02 15:36:57.248875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.248858    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e158e74-74d1-473c-a673-71119444b264-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e158e74-74d1-473c-a673-71119444b264-tls-certs\") pod \"2e158e74-74d1-473c-a673-71119444b264\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") "
Nov 02 15:36:57.249049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.249029    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-model-cache\") pod \"2e158e74-74d1-473c-a673-71119444b264\" (UID: \"2e158e74-74d1-473c-a673-71119444b264\") "
Nov 02 15:36:57.249367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.249070    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-tmp-dir" pod "2e158e74-74d1-473c-a673-71119444b264" (UID: "2e158e74-74d1-473c-a673-71119444b264"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:57.249367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.249249    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-model-cache" pod "2e158e74-74d1-473c-a673-71119444b264" (UID: "2e158e74-74d1-473c-a673-71119444b264"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:57.250046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.250021    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:57.250162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.250072    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:57.250162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.250088    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:57.251445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.251412    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-dshm" pod "2e158e74-74d1-473c-a673-71119444b264" (UID: "2e158e74-74d1-473c-a673-71119444b264"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:36:57.251548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.251502    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e158e74-74d1-473c-a673-71119444b264-tls-certs" pod "2e158e74-74d1-473c-a673-71119444b264" (UID: "2e158e74-74d1-473c-a673-71119444b264"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:36:57.252245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.252213    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e158e74-74d1-473c-a673-71119444b264-kube-api-access-xt7fx" pod "2e158e74-74d1-473c-a673-71119444b264" (UID: "2e158e74-74d1-473c-a673-71119444b264"). InnerVolumeSpecName "kube-api-access-xt7fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:36:57.351604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.351566    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e158e74-74d1-473c-a673-71119444b264-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:57.351604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.351604    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e158e74-74d1-473c-a673-71119444b264-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:57.351604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.351614    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt7fx\" (UniqueName: \"kubernetes.io/projected/2e158e74-74d1-473c-a673-71119444b264-kube-api-access-xt7fx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:36:57.877788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:57.877745    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da823233-c848-48c6-8855-36c6787b174d" path="/var/lib/kubelet/pods/da823233-c848-48c6-8855-36c6787b174d/volumes"
Nov 02 15:36:58.098206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:58.098171    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"
Nov 02 15:36:58.098743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:58.098179    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf" event={"ID":"2e158e74-74d1-473c-a673-71119444b264","Type":"ContainerDied","Data":"273d4896a491cddf49946e8791af400882dd75b82cec8c32ba2ae6d1585a01fb"}
Nov 02 15:36:58.098743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:58.098311    7019 scope.go:122] "RemoveContainer" containerID="9a78dfbf474adc70dae8a8e167524c8fe6fc7755ba59b9e092871032c3d7f395"
Nov 02 15:36:58.169024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:58.168907    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"]
Nov 02 15:36:58.174792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:58.174759    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d35a86/test-model-d35a86-kserve-7988945bf-cmzbf"]
Nov 02 15:36:58.428724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:58.428425    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:36:59.119315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:59.119271    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"]
Nov 02 15:36:59.143684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:59.143634    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01de903b-zv248"]
Nov 02 15:36:59.865695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:59.865657    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e158e74-74d1-473c-a673-71119444b264" path="/var/lib/kubelet/pods/2e158e74-74d1-473c-a673-71119444b264/volumes"
Nov 02 15:36:59.866200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:59.866185    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef69a49-ec78-4a4e-8063-aff678a9ccb1" path="/var/lib/kubelet/pods/aef69a49-ec78-4a4e-8063-aff678a9ccb1/volumes"
Nov 02 15:36:59.952793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:36:59.952748    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v" podUID="1ee2b4d6-d1f2-4577-9c94-419f89c037b2" containerName="main" containerID="cri-o://0c73119aa11fd528cada619a32c6ceeada8c33cf9ae33b4b6afc694705cd8d65" gracePeriod=45
Nov 02 15:37:00.060810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.060772    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v" podUID="1ee2b4d6-d1f2-4577-9c94-419f89c037b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.177:8000/ready\": dial tcp 10.134.0.177:8000: connect: connection refused"
Nov 02 15:37:00.128141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.128057    7019 generic.go:356] "Generic (PLEG): container finished" podID="1ee2b4d6-d1f2-4577-9c94-419f89c037b2" containerID="0c73119aa11fd528cada619a32c6ceeada8c33cf9ae33b4b6afc694705cd8d65" exitCode=0
Nov 02 15:37:00.128530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.128112    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v" event={"ID":"1ee2b4d6-d1f2-4577-9c94-419f89c037b2","Type":"ContainerDied","Data":"0c73119aa11fd528cada619a32c6ceeada8c33cf9ae33b4b6afc694705cd8d65"}
Nov 02 15:37:00.233318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.214027    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:00.278440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.278413    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:37:00.379987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.379899    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg" podUID="71e23bdb-9879-4fe0-9de0-5f3a1b687c17" containerName="main" containerID="cri-o://f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712" gracePeriod=45
Nov 02 15:37:00.416938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.416900    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tmp-dir\") pod \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") "
Nov 02 15:37:00.417074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.416993    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tls-certs\") pod \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") "
Nov 02 15:37:00.417141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.417125    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-dshm\") pod \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") "
Nov 02 15:37:00.417189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.417139    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tmp-dir" pod "1ee2b4d6-d1f2-4577-9c94-419f89c037b2" (UID: "1ee2b4d6-d1f2-4577-9c94-419f89c037b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.417238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.417221    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-model-cache\") pod \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") "
Nov 02 15:37:00.417326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.417310    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-home\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-home\") pod \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") "
Nov 02 15:37:00.417435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.417416    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-kube-api-access-m7lfn\" (UniqueName: \"kubernetes.io/projected/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-kube-api-access-m7lfn\") pod \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\" (UID: \"1ee2b4d6-d1f2-4577-9c94-419f89c037b2\") "
Nov 02 15:37:00.417718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.417533    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-model-cache" pod "1ee2b4d6-d1f2-4577-9c94-419f89c037b2" (UID: "1ee2b4d6-d1f2-4577-9c94-419f89c037b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.417806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.417606    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-home" pod "1ee2b4d6-d1f2-4577-9c94-419f89c037b2" (UID: "1ee2b4d6-d1f2-4577-9c94-419f89c037b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.418529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.418502    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.418625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.418535    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.418625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.418550    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.419520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.419498    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tls-certs" pod "1ee2b4d6-d1f2-4577-9c94-419f89c037b2" (UID: "1ee2b4d6-d1f2-4577-9c94-419f89c037b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:00.419713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.419695    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-kube-api-access-m7lfn" pod "1ee2b4d6-d1f2-4577-9c94-419f89c037b2" (UID: "1ee2b4d6-d1f2-4577-9c94-419f89c037b2"). InnerVolumeSpecName "kube-api-access-m7lfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:00.419929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.419907    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-dshm" pod "1ee2b4d6-d1f2-4577-9c94-419f89c037b2" (UID: "1ee2b4d6-d1f2-4577-9c94-419f89c037b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.519858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.519822    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7lfn\" (UniqueName: \"kubernetes.io/projected/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-kube-api-access-m7lfn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.519858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.519860    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.520077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.519876    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ee2b4d6-d1f2-4577-9c94-419f89c037b2-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.651939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.651911    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:37:00.823750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.823707    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-kube-api-access-79sgc\" (UniqueName: \"kubernetes.io/projected/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-kube-api-access-79sgc\") pod \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") "
Nov 02 15:37:00.823923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.823772    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-home\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-home\") pod \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") "
Nov 02 15:37:00.823923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.823833    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-model-cache\") pod \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") "
Nov 02 15:37:00.823923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.823897    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tmp-dir\") pod \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") "
Nov 02 15:37:00.824074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824014    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tls-certs\" (UniqueName: \"kubernetes.io/secret/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tls-certs\") pod \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") "
Nov 02 15:37:00.824136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824092    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-home" pod "71e23bdb-9879-4fe0-9de0-5f3a1b687c17" (UID: "71e23bdb-9879-4fe0-9de0-5f3a1b687c17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.824200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824152    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-model-cache" pod "71e23bdb-9879-4fe0-9de0-5f3a1b687c17" (UID: "71e23bdb-9879-4fe0-9de0-5f3a1b687c17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.824275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824236    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-dshm\") pod \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\" (UID: \"71e23bdb-9879-4fe0-9de0-5f3a1b687c17\") "
Nov 02 15:37:00.824340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824254    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tmp-dir" pod "71e23bdb-9879-4fe0-9de0-5f3a1b687c17" (UID: "71e23bdb-9879-4fe0-9de0-5f3a1b687c17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.824923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824902    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.825050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824924    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.825050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.824938    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.826180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.826158    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-kube-api-access-79sgc" pod "71e23bdb-9879-4fe0-9de0-5f3a1b687c17" (UID: "71e23bdb-9879-4fe0-9de0-5f3a1b687c17"). InnerVolumeSpecName "kube-api-access-79sgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:00.826274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.826193    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tls-certs" pod "71e23bdb-9879-4fe0-9de0-5f3a1b687c17" (UID: "71e23bdb-9879-4fe0-9de0-5f3a1b687c17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:00.826460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.826432    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-dshm" pod "71e23bdb-9879-4fe0-9de0-5f3a1b687c17" (UID: "71e23bdb-9879-4fe0-9de0-5f3a1b687c17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:00.925928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.925845    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.925928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.925876    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:00.925928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:00.925889    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79sgc\" (UniqueName: \"kubernetes.io/projected/71e23bdb-9879-4fe0-9de0-5f3a1b687c17-kube-api-access-79sgc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:01.139143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.139109    7019 generic.go:356] "Generic (PLEG): container finished" podID="71e23bdb-9879-4fe0-9de0-5f3a1b687c17" containerID="f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712" exitCode=0
Nov 02 15:37:01.139612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.139159    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"
Nov 02 15:37:01.139612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.139195    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg" event={"ID":"71e23bdb-9879-4fe0-9de0-5f3a1b687c17","Type":"ContainerDied","Data":"f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712"}
Nov 02 15:37:01.139612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.139242    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg" event={"ID":"71e23bdb-9879-4fe0-9de0-5f3a1b687c17","Type":"ContainerDied","Data":"7227fea5470a2d98e36fbc932d5fe0afcfcf6f37b159b576246d5a58cb620c55"}
Nov 02 15:37:01.139612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.139265    7019 scope.go:122] "RemoveContainer" containerID="f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712"
Nov 02 15:37:01.141209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.141181    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v" event={"ID":"1ee2b4d6-d1f2-4577-9c94-419f89c037b2","Type":"ContainerDied","Data":"46edefc9e9042764b9f33decb0de169bae76b855724b3d693e83ca667e8d2b11"}
Nov 02 15:37:01.141344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.141220    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"
Nov 02 15:37:01.147546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.147520    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" podUID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" containerName="payload-processing" containerID="cri-o://f0a120c25e68f981657826ecf66408f122918eb58cd338b1a83d282de884edcb" gracePeriod=25
Nov 02 15:37:01.152668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.152593    7019 scope.go:122] "RemoveContainer" containerID="f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712"
Nov 02 15:37:01.153829 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:01.153798    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712\": container with ID starting with f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712 not found: ID does not exist" containerID="f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712"
Nov 02 15:37:01.153934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.153841    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712"} err="failed to get container status \"f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712\": rpc error: code = NotFound desc = could not find container \"f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712\": container with ID starting with f4dd12a1ddac9868b839d7a451189a9d8434226ffc2810529ea0ba0373805712 not found: ID does not exist"
Nov 02 15:37:01.153934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.153868    7019 scope.go:122] "RemoveContainer" containerID="0c73119aa11fd528cada619a32c6ceeada8c33cf9ae33b4b6afc694705cd8d65"
Nov 02 15:37:01.196843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.196771    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"]
Nov 02 15:37:01.209624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.209589    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-69ff88/test-model-69ff88-kserve-967979dfc-hvv8v"]
Nov 02 15:37:01.247774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.247741    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"]
Nov 02 15:37:01.253187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.253151    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-31c816/test-model-31c816-kserve-6c6849585f-qxpjg"]
Nov 02 15:37:01.881508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.880631    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:01.899943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.899908    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee2b4d6-d1f2-4577-9c94-419f89c037b2" path="/var/lib/kubelet/pods/1ee2b4d6-d1f2-4577-9c94-419f89c037b2/volumes"
Nov 02 15:37:01.900626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:01.900598    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e23bdb-9879-4fe0-9de0-5f3a1b687c17" path="/var/lib/kubelet/pods/71e23bdb-9879-4fe0-9de0-5f3a1b687c17/volumes"
Nov 02 15:37:02.045771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:02.045733    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:02.522245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:02.522199    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:03.605538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:03.605466    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:04.172207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:04.172169    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:04.350807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:04.350756    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:04.560018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:04.559928    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:05.107936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.107900    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"]
Nov 02 15:37:05.108902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.108882    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7a7dc0f1-0a16-41ab-99cb-478903dddeb2" containerName="revoke-keys"
Nov 02 15:37:05.108962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.108904    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7dc0f1-0a16-41ab-99cb-478903dddeb2" containerName="revoke-keys"
Nov 02 15:37:05.108962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.108940    7019 cpu_manager.go:414] "containerMap: removing container" podUID="da823233-c848-48c6-8855-36c6787b174d" containerName="revoke-keys"
Nov 02 15:37:05.108962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.108948    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da823233-c848-48c6-8855-36c6787b174d" containerName="revoke-keys"
Nov 02 15:37:05.109058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.108975    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" containerName="revoke-keys"
Nov 02 15:37:05.109058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.108982    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" containerName="revoke-keys"
Nov 02 15:37:05.109058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.108993    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1ee2b4d6-d1f2-4577-9c94-419f89c037b2" containerName="main"
Nov 02 15:37:05.109058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109000    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee2b4d6-d1f2-4577-9c94-419f89c037b2" containerName="main"
Nov 02 15:37:05.109058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109044    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2e158e74-74d1-473c-a673-71119444b264" containerName="main"
Nov 02 15:37:05.109058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109051    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e158e74-74d1-473c-a673-71119444b264" containerName="main"
Nov 02 15:37:05.109242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109065    7019 cpu_manager.go:414] "containerMap: removing container" podUID="71e23bdb-9879-4fe0-9de0-5f3a1b687c17" containerName="main"
Nov 02 15:37:05.109242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109072    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e23bdb-9879-4fe0-9de0-5f3a1b687c17" containerName="main"
Nov 02 15:37:05.109242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109209    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e158e74-74d1-473c-a673-71119444b264" containerName="main"
Nov 02 15:37:05.109242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109223    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e23bdb-9879-4fe0-9de0-5f3a1b687c17" containerName="main"
Nov 02 15:37:05.109242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109236    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7dc0f1-0a16-41ab-99cb-478903dddeb2" containerName="revoke-keys"
Nov 02 15:37:05.109402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109249    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="da823233-c848-48c6-8855-36c6787b174d" containerName="revoke-keys"
Nov 02 15:37:05.109402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109263    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" containerName="revoke-keys"
Nov 02 15:37:05.109402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.109272    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee2b4d6-d1f2-4577-9c94-419f89c037b2" containerName="main"
Nov 02 15:37:05.114810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.114783    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.139496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.139466    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"]
Nov 02 15:37:05.185427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.185393    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw27b\" (UniqueName: \"kubernetes.io/projected/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-kube-api-access-lw27b\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.185636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.185442    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.185636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.185461    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.212418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.212387    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:37:05.212586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.212464    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:37:05.287373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.287336    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw27b\" (UniqueName: \"kubernetes.io/projected/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-kube-api-access-lw27b\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.287545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.287389    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.287545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.287503    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.288157 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.288140    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.290146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.290124    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.319619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.319583    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw27b\" (UniqueName: \"kubernetes.io/projected/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-kube-api-access-lw27b\") pod \"maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.428931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.428900    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:05.579721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.577264    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:05.742316 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:37:05.742285    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e4fcd4_79fb_4511_ba0a_3c3a6e6f8340.slice/crio-2dd79fd9b48afbb11e8fd1d8c190ec9cf3d9e32a432557f4c84f3852e36d28b1 WatchSource:0}: Error finding container 2dd79fd9b48afbb11e8fd1d8c190ec9cf3d9e32a432557f4c84f3852e36d28b1: Status 404 returned error can't find the container with id 2dd79fd9b48afbb11e8fd1d8c190ec9cf3d9e32a432557f4c84f3852e36d28b1
Nov 02 15:37:05.745247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.745219    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"]
Nov 02 15:37:05.969991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:05.969948    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" podUID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:06.188484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:06.188441    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd" event={"ID":"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340","Type":"ContainerStarted","Data":"437c54635cd4d9b6632f12b3700fefe5055b3765bc5be08ac1d309e543f40521"}
Nov 02 15:37:06.188928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:06.188506    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd" event={"ID":"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340","Type":"ContainerStarted","Data":"2dd79fd9b48afbb11e8fd1d8c190ec9cf3d9e32a432557f4c84f3852e36d28b1"}
Nov 02 15:37:06.233062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:06.233005    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd" podStartSLOduration=1.232990574 podStartE2EDuration="1.232990574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:06.231163715 +0000 UTC m=+2420.485117367" watchObservedRunningTime="2026-11-02 15:37:06.232990574 +0000 UTC m=+2420.486944197"
Nov 02 15:37:07.198799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:07.198757    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:07.199460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:07.199436    7019 generic.go:356] "Generic (PLEG): container finished" podID="48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" containerID="437c54635cd4d9b6632f12b3700fefe5055b3765bc5be08ac1d309e543f40521" exitCode=0
Nov 02 15:37:07.199543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:07.199517    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd" event={"ID":"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340","Type":"ContainerDied","Data":"437c54635cd4d9b6632f12b3700fefe5055b3765bc5be08ac1d309e543f40521"}
Nov 02 15:37:08.388399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.388367    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:08.446120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.446077    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-kube-api-access-lw27b\" (UniqueName: \"kubernetes.io/projected/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-kube-api-access-lw27b\") pod \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") "
Nov 02 15:37:08.446305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.446208    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-xks-ca\") pod \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") "
Nov 02 15:37:08.446400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.446381    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-openshift-ca\") pod \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\" (UID: \"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340\") "
Nov 02 15:37:08.446907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.446881    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-openshift-ca" pod "48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" (UID: "48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:08.447290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.447263    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:08.448757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.448729    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-xks-ca" pod "48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" (UID: "48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:08.448757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.448746    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-kube-api-access-lw27b" pod "48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" (UID: "48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340"). InnerVolumeSpecName "kube-api-access-lw27b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:08.549178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.549117    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw27b\" (UniqueName: \"kubernetes.io/projected/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-kube-api-access-lw27b\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:08.549178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.549161    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:08.770883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:08.770780    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:09.252857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.252810    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd" event={"ID":"48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340","Type":"ContainerDied","Data":"2dd79fd9b48afbb11e8fd1d8c190ec9cf3d9e32a432557f4c84f3852e36d28b1"}
Nov 02 15:37:09.253153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.253132    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd79fd9b48afbb11e8fd1d8c190ec9cf3d9e32a432557f4c84f3852e36d28b1"
Nov 02 15:37:09.253331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.253315    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"
Nov 02 15:37:09.365335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.365297    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:09.465707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.465634    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"]
Nov 02 15:37:09.469078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.469050    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cfc63a-vm9lb"]
Nov 02 15:37:09.523519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.523357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:09.864977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:09.864948    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978d43d5-e5d0-4f6e-8727-aadb6fa066ab" path="/var/lib/kubelet/pods/978d43d5-e5d0-4f6e-8727-aadb6fa066ab/volumes"
Nov 02 15:37:10.091522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:10.091486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:11.140207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:11.140173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:11.656953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:11.656910    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:11.767369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:11.767332    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"]
Nov 02 15:37:11.785440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:11.785404    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-80ec03-4d4k8"]
Nov 02 15:37:11.825672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:11.815221    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:11.863776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:11.863745    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1" path="/var/lib/kubelet/pods/1a81c2bf-82f1-448f-8bd8-df6bc7f9b7b1/volumes"
Nov 02 15:37:12.314419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:12.314383    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:13.707678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:13.705308    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:15.456111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:15.451186    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:15.689455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:15.689420    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"]
Nov 02 15:37:15.840681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:15.840542    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:15.969542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:15.969500    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" podUID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:16.019115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.019070    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:16.283120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.283083    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"]
Nov 02 15:37:16.374229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.374162    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:16.580285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.580190    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"]
Nov 02 15:37:16.582466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.582409    7019 cpu_manager.go:414] "containerMap: removing container" podUID="48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" containerName="revoke-keys"
Nov 02 15:37:16.582642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.582627    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" containerName="revoke-keys"
Nov 02 15:37:16.583497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.583476    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" containerName="revoke-keys"
Nov 02 15:37:16.598059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.596963    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.609599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.609572    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"]
Nov 02 15:37:16.620667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.617993    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:37:16.620667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.618251    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:37:16.662695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.655816    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94p5h\" (UniqueName: \"kubernetes.io/projected/4e534392-4512-45fd-bf64-c678d5f14b70-kube-api-access-94p5h\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.662695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.655889    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.662695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.655943    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.757759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.757725    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.758017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.757993    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94p5h\" (UniqueName: \"kubernetes.io/projected/4e534392-4512-45fd-bf64-c678d5f14b70-kube-api-access-94p5h\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.758104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.758088    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.758760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.758737    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.760165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.760145    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.783789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.783758    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94p5h\" (UniqueName: \"kubernetes.io/projected/4e534392-4512-45fd-bf64-c678d5f14b70-kube-api-access-94p5h\") pod \"maas-api-revoke-keys-e2e-inf-a-69ff88-p284h\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:16.930129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:16.930098    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:17.247109 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:37:17.247080    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e534392_4512_45fd_bf64_c678d5f14b70.slice/crio-dc8c0323feb5850c2b9b87487b973e185fdc3d82f738a1b6c807861831160961 WatchSource:0}: Error finding container dc8c0323feb5850c2b9b87487b973e185fdc3d82f738a1b6c807861831160961: Status 404 returned error can't find the container with id dc8c0323feb5850c2b9b87487b973e185fdc3d82f738a1b6c807861831160961
Nov 02 15:37:17.249383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.249352    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"]
Nov 02 15:37:17.335429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.335382    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h" event={"ID":"4e534392-4512-45fd-bf64-c678d5f14b70","Type":"ContainerStarted","Data":"dc8c0323feb5850c2b9b87487b973e185fdc3d82f738a1b6c807861831160961"}
Nov 02 15:37:17.337131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.337099    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:17.668965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.668929    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:17.869464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.869430    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:17.898880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.891889    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"]
Nov 02 15:37:17.898880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.892144    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" containerID="cri-o://96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4" gracePeriod=30
Nov 02 15:37:17.910022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.909909    7019 patch_prober.go:29] interesting pod/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:37:17.912763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:17.912732    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:37:18.080724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:18.076384    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:18.345679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:18.345554    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h" event={"ID":"4e534392-4512-45fd-bf64-c678d5f14b70","Type":"ContainerStarted","Data":"dc9f5728bb67ca96eed86a51925ee38f7c87c67a52fdbe635958bb6d764b52b5"}
Nov 02 15:37:18.392217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:18.392162    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h" podStartSLOduration=2.392148016 podStartE2EDuration="2.392148016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:18.388163688 +0000 UTC m=+2432.642117314" watchObservedRunningTime="2026-11-02 15:37:18.392148016 +0000 UTC m=+2432.646101634"
Nov 02 15:37:19.187670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:19.185347    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:19.355269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:19.355234    7019 generic.go:356] "Generic (PLEG): container finished" podID="4e534392-4512-45fd-bf64-c678d5f14b70" containerID="dc9f5728bb67ca96eed86a51925ee38f7c87c67a52fdbe635958bb6d764b52b5" exitCode=0
Nov 02 15:37:19.355462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:19.355312    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h" event={"ID":"4e534392-4512-45fd-bf64-c678d5f14b70","Type":"ContainerDied","Data":"dc9f5728bb67ca96eed86a51925ee38f7c87c67a52fdbe635958bb6d764b52b5"}
Nov 02 15:37:19.464488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:19.464391    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:19.641080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:19.641046    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:19.835543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:19.835441    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:20.387582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.387546    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"]
Nov 02 15:37:20.393278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.393253    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.414988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.414963    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"]
Nov 02 15:37:20.463804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.463766    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.464010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.463981    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.464082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.464051    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvpt\" (UniqueName: \"kubernetes.io/projected/55adfe75-2cdb-4356-8dde-212a00ac539e-kube-api-access-7mvpt\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.565886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.565851    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.566059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.566038    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.566126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.566104    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvpt\" (UniqueName: \"kubernetes.io/projected/55adfe75-2cdb-4356-8dde-212a00ac539e-kube-api-access-7mvpt\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.566858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.566835    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.568333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.568308    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.589128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.589101    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:20.591113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.591084    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvpt\" (UniqueName: \"kubernetes.io/projected/55adfe75-2cdb-4356-8dde-212a00ac539e-kube-api-access-7mvpt\") pod \"maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.666905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.666810    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-openshift-ca\") pod \"4e534392-4512-45fd-bf64-c678d5f14b70\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") "
Nov 02 15:37:20.666905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.666882    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e534392-4512-45fd-bf64-c678d5f14b70-kube-api-access-94p5h\" (UniqueName: \"kubernetes.io/projected/4e534392-4512-45fd-bf64-c678d5f14b70-kube-api-access-94p5h\") pod \"4e534392-4512-45fd-bf64-c678d5f14b70\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") "
Nov 02 15:37:20.667345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.666931    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-xks-ca\") pod \"4e534392-4512-45fd-bf64-c678d5f14b70\" (UID: \"4e534392-4512-45fd-bf64-c678d5f14b70\") "
Nov 02 15:37:20.667345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.667971    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-openshift-ca" pod "4e534392-4512-45fd-bf64-c678d5f14b70" (UID: "4e534392-4512-45fd-bf64-c678d5f14b70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:20.668634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.668590    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:20.670488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.670454    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e534392-4512-45fd-bf64-c678d5f14b70-kube-api-access-94p5h" pod "4e534392-4512-45fd-bf64-c678d5f14b70" (UID: "4e534392-4512-45fd-bf64-c678d5f14b70"). InnerVolumeSpecName "kube-api-access-94p5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:20.670756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.670715    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-xks-ca" pod "4e534392-4512-45fd-bf64-c678d5f14b70" (UID: "4e534392-4512-45fd-bf64-c678d5f14b70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:20.690715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.690672    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" podUID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" containerName="payload-pre-processing" containerID="cri-o://479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd" gracePeriod=25
Nov 02 15:37:20.714467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.714440    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:20.770696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.770584    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94p5h\" (UniqueName: \"kubernetes.io/projected/4e534392-4512-45fd-bf64-c678d5f14b70-kube-api-access-94p5h\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:20.770696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:20.770617    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e534392-4512-45fd-bf64-c678d5f14b70-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:21.033536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.033426    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"]
Nov 02 15:37:21.044176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.044139    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:21.044835 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:37:21.044795    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55adfe75_2cdb_4356_8dde_212a00ac539e.slice/crio-4fd99960507665ba538f456461417ac2b46bee0736e277dacff4e64f7041a179 WatchSource:0}: Error finding container 4fd99960507665ba538f456461417ac2b46bee0736e277dacff4e64f7041a179: Status 404 returned error can't find the container with id 4fd99960507665ba538f456461417ac2b46bee0736e277dacff4e64f7041a179
Nov 02 15:37:21.382491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.382034    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc" event={"ID":"55adfe75-2cdb-4356-8dde-212a00ac539e","Type":"ContainerStarted","Data":"4fd99960507665ba538f456461417ac2b46bee0736e277dacff4e64f7041a179"}
Nov 02 15:37:21.384949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.384895    7019 generic.go:356] "Generic (PLEG): container finished" podID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" containerID="f0a120c25e68f981657826ecf66408f122918eb58cd338b1a83d282de884edcb" exitCode=0
Nov 02 15:37:21.385262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.385228    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" event={"ID":"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64","Type":"ContainerDied","Data":"f0a120c25e68f981657826ecf66408f122918eb58cd338b1a83d282de884edcb"}
Nov 02 15:37:21.387601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.387579    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h" event={"ID":"4e534392-4512-45fd-bf64-c678d5f14b70","Type":"ContainerDied","Data":"dc8c0323feb5850c2b9b87487b973e185fdc3d82f738a1b6c807861831160961"}
Nov 02 15:37:21.388117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.387620    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc8c0323feb5850c2b9b87487b973e185fdc3d82f738a1b6c807861831160961"
Nov 02 15:37:21.388117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.387660    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"
Nov 02 15:37:21.434917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.434885    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:37:21.480996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.480968    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-plugins-config-volume\") pod \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") "
Nov 02 15:37:21.481135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.481094    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-kube-api-access-p5z7s\" (UniqueName: \"kubernetes.io/projected/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-kube-api-access-p5z7s\") pod \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\" (UID: \"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64\") "
Nov 02 15:37:21.481355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.481334    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-plugins-config-volume" pod "11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" (UID: "11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:21.481951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.481924    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:21.483075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.483052    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-kube-api-access-p5z7s" pod "11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" (UID: "11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64"). InnerVolumeSpecName "kube-api-access-p5z7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:21.585258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.583323    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5z7s\" (UniqueName: \"kubernetes.io/projected/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64-kube-api-access-p5z7s\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:21.781209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:21.781171    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" podUID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:22.397580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.397542    7019 generic.go:356] "Generic (PLEG): container finished" podID="55adfe75-2cdb-4356-8dde-212a00ac539e" containerID="d9a56ab69411d75ad8bba70567a08236e2d99162977209957da2729c78ecbcda" exitCode=0
Nov 02 15:37:22.398108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.397633    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc" event={"ID":"55adfe75-2cdb-4356-8dde-212a00ac539e","Type":"ContainerDied","Data":"d9a56ab69411d75ad8bba70567a08236e2d99162977209957da2729c78ecbcda"}
Nov 02 15:37:22.399550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.399526    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"
Nov 02 15:37:22.399779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.399751    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr" event={"ID":"11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64","Type":"ContainerDied","Data":"3c5e759d5c4d012114709780bb4659e6063ee68e316e4a28fd0c29e67491ffb4"}
Nov 02 15:37:22.399871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.399829    7019 scope.go:122] "RemoveContainer" containerID="f0a120c25e68f981657826ecf66408f122918eb58cd338b1a83d282de884edcb"
Nov 02 15:37:22.483534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.483447    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"]
Nov 02 15:37:22.495419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.495385    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80ec03-58fc7d44c9-9grgr"]
Nov 02 15:37:22.711905 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:22.711853    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:37:22.943431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:22.939500    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:23.188957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.188886    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:37:23.308001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.307966    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-token\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308081    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-podinfo\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308115    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-credential-socket\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308156    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-data\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308211    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istiod-ca-cert\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308263    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-kube-api-access-jrr24\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-kube-api-access-jrr24\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308301    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-socket\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308355    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-envoy\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308556    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-certs\") pod \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\" (UID: \"d6d85dc4-dad6-4112-a6d0-ca4f29074444\") "
Nov 02 15:37:23.308699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.308628    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-data" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:23.309138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.309114    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.309528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.309500    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-socket" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:23.309956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.309931    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istiod-ca-cert" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:23.310561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.310534    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-credential-socket" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:23.310785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.310762    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-certs" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:23.311524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.311274    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-token" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:23.312017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.311989    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-kube-api-access-jrr24" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "kube-api-access-jrr24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:23.312592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.312563    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-podinfo" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:23.315963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.315940    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-envoy" pod "d6d85dc4-dad6-4112-a6d0-ca4f29074444" (UID: "d6d85dc4-dad6-4112-a6d0-ca4f29074444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:23.410798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410767    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.410798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410797    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.411347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410810    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.411347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410824    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrr24\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-kube-api-access-jrr24\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.411347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410839    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.411347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410851    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.411347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410862    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d85dc4-dad6-4112-a6d0-ca4f29074444-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.411347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.410874    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d85dc4-dad6-4112-a6d0-ca4f29074444-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.412439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.412405    7019 generic.go:356] "Generic (PLEG): container finished" podID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerID="96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4" exitCode=0
Nov 02 15:37:23.412583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.412468    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"
Nov 02 15:37:23.412583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.412476    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" event={"ID":"d6d85dc4-dad6-4112-a6d0-ca4f29074444","Type":"ContainerDied","Data":"96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4"}
Nov 02 15:37:23.412583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.412533    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n" event={"ID":"d6d85dc4-dad6-4112-a6d0-ca4f29074444","Type":"ContainerDied","Data":"168c55230878287b07ec68b3ca8bcc37dba1dc7c1107e1f50f06308306505640"}
Nov 02 15:37:23.412583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.412563    7019 scope.go:122] "RemoveContainer" containerID="96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4"
Nov 02 15:37:23.425565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.425536    7019 scope.go:122] "RemoveContainer" containerID="39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73"
Nov 02 15:37:23.437379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.437348    7019 scope.go:122] "RemoveContainer" containerID="96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4"
Nov 02 15:37:23.437729 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:23.437703    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4\": container with ID starting with 96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4 not found: ID does not exist" containerID="96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4"
Nov 02 15:37:23.437828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.437736    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4"} err="failed to get container status \"96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4\": rpc error: code = NotFound desc = could not find container \"96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4\": container with ID starting with 96adbb67a6e0587b342e4858dd04ec659739bec02c7b7c4d6ef6a9640ccbbef4 not found: ID does not exist"
Nov 02 15:37:23.437828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.437757    7019 scope.go:122] "RemoveContainer" containerID="39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73"
Nov 02 15:37:23.438059 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:23.438039    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73\": container with ID starting with 39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73 not found: ID does not exist" containerID="39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73"
Nov 02 15:37:23.438145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.438062    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73"} err="failed to get container status \"39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73\": rpc error: code = NotFound desc = could not find container \"39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73\": container with ID starting with 39bc4e4202029efcbd011de4ea4fd8b47f0734a235028c8fd78691f4e4a09a73 not found: ID does not exist"
Nov 02 15:37:23.475046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.474953    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"]
Nov 02 15:37:23.487186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.479165    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cfc63a-openshift-default-567d7f7b85-jwg8n"]
Nov 02 15:37:23.584555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.584527    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:23.716051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.716013    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-openshift-ca\") pod \"55adfe75-2cdb-4356-8dde-212a00ac539e\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") "
Nov 02 15:37:23.716225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.716064    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55adfe75-2cdb-4356-8dde-212a00ac539e-kube-api-access-7mvpt\" (UniqueName: \"kubernetes.io/projected/55adfe75-2cdb-4356-8dde-212a00ac539e-kube-api-access-7mvpt\") pod \"55adfe75-2cdb-4356-8dde-212a00ac539e\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") "
Nov 02 15:37:23.716225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.716095    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-xks-ca\") pod \"55adfe75-2cdb-4356-8dde-212a00ac539e\" (UID: \"55adfe75-2cdb-4356-8dde-212a00ac539e\") "
Nov 02 15:37:23.716420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.716396    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-openshift-ca" pod "55adfe75-2cdb-4356-8dde-212a00ac539e" (UID: "55adfe75-2cdb-4356-8dde-212a00ac539e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:23.716716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.716698    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.718317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.718292    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-xks-ca" pod "55adfe75-2cdb-4356-8dde-212a00ac539e" (UID: "55adfe75-2cdb-4356-8dde-212a00ac539e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:23.718381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.718306    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55adfe75-2cdb-4356-8dde-212a00ac539e-kube-api-access-7mvpt" pod "55adfe75-2cdb-4356-8dde-212a00ac539e" (UID: "55adfe75-2cdb-4356-8dde-212a00ac539e"). InnerVolumeSpecName "kube-api-access-7mvpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:23.818587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.818493    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvpt\" (UniqueName: \"kubernetes.io/projected/55adfe75-2cdb-4356-8dde-212a00ac539e-kube-api-access-7mvpt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.818587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.818527    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55adfe75-2cdb-4356-8dde-212a00ac539e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:23.871838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.871557    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" path="/var/lib/kubelet/pods/11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64/volumes"
Nov 02 15:37:23.872370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:23.872305    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" path="/var/lib/kubelet/pods/d6d85dc4-dad6-4112-a6d0-ca4f29074444/volumes"
Nov 02 15:37:24.422382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:24.422354    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"
Nov 02 15:37:24.422818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:24.422392    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc" event={"ID":"55adfe75-2cdb-4356-8dde-212a00ac539e","Type":"ContainerDied","Data":"4fd99960507665ba538f456461417ac2b46bee0736e277dacff4e64f7041a179"}
Nov 02 15:37:24.422818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:24.422448    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fd99960507665ba538f456461417ac2b46bee0736e277dacff4e64f7041a179"
Nov 02 15:37:24.731678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:24.731565    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:26.377756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:26.376105    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:26.990706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:26.990662    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"]
Nov 02 15:37:27.164978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:27.164943    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:27.316726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:27.316631    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:28.012674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:28.012125    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:29.724287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:29.705562    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:30.868100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:30.868063    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"]
Nov 02 15:37:30.909537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:30.909500    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"]
Nov 02 15:37:31.079896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.079860    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:31.223979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.223881    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:31.307092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.307049    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd" podUID="a6345111-1120-4afd-af75-733bae7aa495" containerName="main" containerID="cri-o://b2f101466ac9e7a3002c99b9c46a973971ca395a6f75a17fcd8def3aabb84d5f" gracePeriod=45
Nov 02 15:37:31.489114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.489068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:31.490625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.490599    7019 generic.go:356] "Generic (PLEG): container finished" podID="a6345111-1120-4afd-af75-733bae7aa495" containerID="b2f101466ac9e7a3002c99b9c46a973971ca395a6f75a17fcd8def3aabb84d5f" exitCode=0
Nov 02 15:37:31.490842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.490794    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd" event={"ID":"a6345111-1120-4afd-af75-733bae7aa495","Type":"ContainerDied","Data":"b2f101466ac9e7a3002c99b9c46a973971ca395a6f75a17fcd8def3aabb84d5f"}
Nov 02 15:37:31.758427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.758357    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:37:31.781360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.781324    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" podUID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:31.785935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.785901    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-home\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-home\") pod \"a6345111-1120-4afd-af75-733bae7aa495\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") "
Nov 02 15:37:31.786118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.785969    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-tmp-dir\") pod \"a6345111-1120-4afd-af75-733bae7aa495\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") "
Nov 02 15:37:31.786118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.786040    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6345111-1120-4afd-af75-733bae7aa495-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6345111-1120-4afd-af75-733bae7aa495-tls-certs\") pod \"a6345111-1120-4afd-af75-733bae7aa495\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") "
Nov 02 15:37:31.786240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.786156    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-dshm\") pod \"a6345111-1120-4afd-af75-733bae7aa495\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") "
Nov 02 15:37:31.786240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.786183    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-home" pod "a6345111-1120-4afd-af75-733bae7aa495" (UID: "a6345111-1120-4afd-af75-733bae7aa495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:31.786330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.786290    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-tmp-dir" pod "a6345111-1120-4afd-af75-733bae7aa495" (UID: "a6345111-1120-4afd-af75-733bae7aa495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:31.786489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.786464    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6345111-1120-4afd-af75-733bae7aa495-kube-api-access-6zdfm\" (UniqueName: \"kubernetes.io/projected/a6345111-1120-4afd-af75-733bae7aa495-kube-api-access-6zdfm\") pod \"a6345111-1120-4afd-af75-733bae7aa495\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") "
Nov 02 15:37:31.786579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.786529    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-model-cache\") pod \"a6345111-1120-4afd-af75-733bae7aa495\" (UID: \"a6345111-1120-4afd-af75-733bae7aa495\") "
Nov 02 15:37:31.787117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.787091    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-model-cache" pod "a6345111-1120-4afd-af75-733bae7aa495" (UID: "a6345111-1120-4afd-af75-733bae7aa495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:31.787387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.787361    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:31.787711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.787390    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:31.787711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.787404    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:31.788622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.788594    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6345111-1120-4afd-af75-733bae7aa495-tls-certs" pod "a6345111-1120-4afd-af75-733bae7aa495" (UID: "a6345111-1120-4afd-af75-733bae7aa495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:31.788809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.788787    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-dshm" pod "a6345111-1120-4afd-af75-733bae7aa495" (UID: "a6345111-1120-4afd-af75-733bae7aa495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:31.788918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.788896    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6345111-1120-4afd-af75-733bae7aa495-kube-api-access-6zdfm" pod "a6345111-1120-4afd-af75-733bae7aa495" (UID: "a6345111-1120-4afd-af75-733bae7aa495"). InnerVolumeSpecName "kube-api-access-6zdfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:31.889355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.889314    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zdfm\" (UniqueName: \"kubernetes.io/projected/a6345111-1120-4afd-af75-733bae7aa495-kube-api-access-6zdfm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:31.889355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.889355    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6345111-1120-4afd-af75-733bae7aa495-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:31.889892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.889371    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6345111-1120-4afd-af75-733bae7aa495-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:31.991600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:31.991557    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" podUID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" containerName="payload-processing" containerID="cri-o://1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1" gracePeriod=25
Nov 02 15:37:32.534404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:32.530187    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd" event={"ID":"a6345111-1120-4afd-af75-733bae7aa495","Type":"ContainerDied","Data":"c04b96f31b2548152c530ccb7e28b4eec67fc564db955b6d9910970cafa25ea9"}
Nov 02 15:37:32.534404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:32.530248    7019 scope.go:122] "RemoveContainer" containerID="b2f101466ac9e7a3002c99b9c46a973971ca395a6f75a17fcd8def3aabb84d5f"
Nov 02 15:37:32.534404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:32.530421    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"
Nov 02 15:37:32.691480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:32.691441    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"]
Nov 02 15:37:32.700798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:32.700764    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2e4ebc/test-model-2e4ebc-kserve-55bb8f569c-z2htd"]
Nov 02 15:37:33.442329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:33.442247    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:33.863101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:33.863072    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6345111-1120-4afd-af75-733bae7aa495" path="/var/lib/kubelet/pods/a6345111-1120-4afd-af75-733bae7aa495/volumes"
Nov 02 15:37:34.935179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:34.935143    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:35.453376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:35.453342    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:35.550347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:35.550310    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"]
Nov 02 15:37:35.563310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:35.563265    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-69ff88-p284h"]
Nov 02 15:37:35.630309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:35.630271    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:35.862775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:35.862746    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e534392-4512-45fd-bf64-c678d5f14b70" path="/var/lib/kubelet/pods/4e534392-4512-45fd-bf64-c678d5f14b70/volumes"
Nov 02 15:37:35.869207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:35.869180    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx" podUID="c9009810-8214-4b24-bbad-e86b2575ac41" containerName="payload-processing" containerID="cri-o://e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925" gracePeriod=25
Nov 02 15:37:35.910724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:35.910694    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" podUID="143c471f-15dc-47a0-b7d3-f10557a08f0a" containerName="payload-pre-processing" containerID="cri-o://37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91" gracePeriod=25
Nov 02 15:37:36.094466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.094424    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:36.143989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.143966    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:37:36.270843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.270805    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9009810-8214-4b24-bbad-e86b2575ac41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9009810-8214-4b24-bbad-e86b2575ac41-plugins-config-volume\") pod \"c9009810-8214-4b24-bbad-e86b2575ac41\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") "
Nov 02 15:37:36.271048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.270890    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9009810-8214-4b24-bbad-e86b2575ac41-kube-api-access-8xpv2\" (UniqueName: \"kubernetes.io/projected/c9009810-8214-4b24-bbad-e86b2575ac41-kube-api-access-8xpv2\") pod \"c9009810-8214-4b24-bbad-e86b2575ac41\" (UID: \"c9009810-8214-4b24-bbad-e86b2575ac41\") "
Nov 02 15:37:36.271185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.271158    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9009810-8214-4b24-bbad-e86b2575ac41-plugins-config-volume" pod "c9009810-8214-4b24-bbad-e86b2575ac41" (UID: "c9009810-8214-4b24-bbad-e86b2575ac41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:36.271630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.271610    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9009810-8214-4b24-bbad-e86b2575ac41-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:36.273231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.273208    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9009810-8214-4b24-bbad-e86b2575ac41-kube-api-access-8xpv2" pod "c9009810-8214-4b24-bbad-e86b2575ac41" (UID: "c9009810-8214-4b24-bbad-e86b2575ac41"). InnerVolumeSpecName "kube-api-access-8xpv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:36.373512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.373430    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xpv2\" (UniqueName: \"kubernetes.io/projected/c9009810-8214-4b24-bbad-e86b2575ac41-kube-api-access-8xpv2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:36.521317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.521278    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" podUID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:36.577629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.577590    7019 generic.go:356] "Generic (PLEG): container finished" podID="c9009810-8214-4b24-bbad-e86b2575ac41" containerID="e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925" exitCode=0
Nov 02 15:37:36.577819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.577637    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"
Nov 02 15:37:36.577819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.577679    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx" event={"ID":"c9009810-8214-4b24-bbad-e86b2575ac41","Type":"ContainerDied","Data":"e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925"}
Nov 02 15:37:36.577819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.577739    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx" event={"ID":"c9009810-8214-4b24-bbad-e86b2575ac41","Type":"ContainerDied","Data":"2d14e3cffe166def85100cf88769a53a10d5ab58c78e9a8e60b296f9f454a9a9"}
Nov 02 15:37:36.577819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.577762    7019 scope.go:122] "RemoveContainer" containerID="e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925"
Nov 02 15:37:36.591967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.591943    7019 scope.go:122] "RemoveContainer" containerID="e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925"
Nov 02 15:37:36.592231 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:36.592212    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925\": container with ID starting with e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925 not found: ID does not exist" containerID="e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925"
Nov 02 15:37:36.592311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.592238    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925"} err="failed to get container status \"e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925\": rpc error: code = NotFound desc = could not find container \"e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925\": container with ID starting with e9a209f06c546e9dfb61c58c32b57ef4abe3cb7d119f29c93643e9012d34f925 not found: ID does not exist"
Nov 02 15:37:36.636989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.636897    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"]
Nov 02 15:37:36.640112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:36.640092    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-31c816-cb456bb99-4dfwx"]
Nov 02 15:37:37.596339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:37.596305    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:37.865771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:37.865728    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9009810-8214-4b24-bbad-e86b2575ac41" path="/var/lib/kubelet/pods/c9009810-8214-4b24-bbad-e86b2575ac41/volumes"
Nov 02 15:37:38.622849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:38.622807    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" podUID="143c471f-15dc-47a0-b7d3-f10557a08f0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:39.311434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:39.311394    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:40.862298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:40.862262    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:41.003356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.003320    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:37:41.064590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.064559    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-plugins-config-volume\") pod \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") "
Nov 02 15:37:41.064797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.064766    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-kube-api-access-qdjfp\" (UniqueName: \"kubernetes.io/projected/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-kube-api-access-qdjfp\") pod \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\" (UID: \"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92\") "
Nov 02 15:37:41.065059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.065031    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-plugins-config-volume" pod "e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" (UID: "e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:41.065350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.065329    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:41.067475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.067448    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-kube-api-access-qdjfp" pod "e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" (UID: "e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92"). InnerVolumeSpecName "kube-api-access-qdjfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:41.167152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.167114    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdjfp\" (UniqueName: \"kubernetes.io/projected/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92-kube-api-access-qdjfp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:41.649528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.649495    7019 generic.go:356] "Generic (PLEG): container finished" podID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" containerID="479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd" exitCode=0
Nov 02 15:37:41.649742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.649547    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"
Nov 02 15:37:41.649742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.649578    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" event={"ID":"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92","Type":"ContainerDied","Data":"479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd"}
Nov 02 15:37:41.649742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.649703    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs" event={"ID":"e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92","Type":"ContainerDied","Data":"89af593817e89c93cd322310f6d5ceea88822e40b6c516c7d2f6ec466ab31903"}
Nov 02 15:37:41.649742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.649725    7019 scope.go:122] "RemoveContainer" containerID="479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd"
Nov 02 15:37:41.661448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.661430    7019 scope.go:122] "RemoveContainer" containerID="479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd"
Nov 02 15:37:41.661732 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:41.661710    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd\": container with ID starting with 479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd not found: ID does not exist" containerID="479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd"
Nov 02 15:37:41.661829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.661738    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd"} err="failed to get container status \"479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd\": rpc error: code = NotFound desc = could not find container \"479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd\": container with ID starting with 479d8c51d5871d8a30d6329e98dc7333a980eb878b9d39cdfbd5b5e6b7616bdd not found: ID does not exist"
Nov 02 15:37:41.708135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.708091    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"]
Nov 02 15:37:41.711636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.711609    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d35a86-66bf588b7f-2tcbs"]
Nov 02 15:37:41.862719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:41.862682    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" path="/var/lib/kubelet/pods/e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92/volumes"
Nov 02 15:37:42.500496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:42.500456    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:43.141850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.141816    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:43.299622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.299576    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:43.382404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.382368    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"]
Nov 02 15:37:43.464241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.464156    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:43.667130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.667087    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" containerID="cri-o://9fba8a6787ae0f3bfdaa8ed913a5dc0e7afaefe1f3f47c319f8697a8e25f48b0" gracePeriod=30
Nov 02 15:37:43.921993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.921958    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:43.976832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.976792    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"]
Nov 02 15:37:43.977823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977795    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c9009810-8214-4b24-bbad-e86b2575ac41" containerName="payload-processing"
Nov 02 15:37:43.977823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977819    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9009810-8214-4b24-bbad-e86b2575ac41" containerName="payload-processing"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977838    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4e534392-4512-45fd-bf64-c678d5f14b70" containerName="revoke-keys"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977848    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e534392-4512-45fd-bf64-c678d5f14b70" containerName="revoke-keys"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977873    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" containerName="payload-pre-processing"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977880    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" containerName="payload-pre-processing"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977915    7019 cpu_manager.go:414] "containerMap: removing container" podUID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" containerName="payload-processing"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977923    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" containerName="payload-processing"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977932    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977939    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977958    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977965    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977982    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a6345111-1120-4afd-af75-733bae7aa495" containerName="main"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.977989    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6345111-1120-4afd-af75-733bae7aa495" containerName="main"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978016    7019 cpu_manager.go:414] "containerMap: removing container" podUID="55adfe75-2cdb-4356-8dde-212a00ac539e" containerName="revoke-keys"
Nov 02 15:37:43.978037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978023    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55adfe75-2cdb-4356-8dde-212a00ac539e" containerName="revoke-keys"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978170    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a1ff58-cd58-4663-aa5c-6fbf0d7fcb64" containerName="payload-processing"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978182    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978196    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d85dc4-dad6-4112-a6d0-ca4f29074444" containerName="istio-proxy"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978208    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6345111-1120-4afd-af75-733bae7aa495" containerName="main"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978222    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="55adfe75-2cdb-4356-8dde-212a00ac539e" containerName="revoke-keys"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978235    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9009810-8214-4b24-bbad-e86b2575ac41" containerName="payload-processing"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978243    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e534392-4512-45fd-bf64-c678d5f14b70" containerName="revoke-keys"
Nov 02 15:37:43.978863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.978257    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ab4380-3dc2-49b4-9c9c-f6eba9a44a92" containerName="payload-pre-processing"
Nov 02 15:37:43.982687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.982640    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.001097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:43.999830    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:37:44.001764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.001735    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:37:44.008771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.008734    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"]
Nov 02 15:37:44.142799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.142761    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.152699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.151931    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.152699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.152090    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxgf\" (UniqueName: \"kubernetes.io/projected/4ac86b66-096a-4384-a182-c379dbb54b5d-kube-api-access-5kxgf\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.253963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.253880    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.254145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.254009    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.254145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.254104    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxgf\" (UniqueName: \"kubernetes.io/projected/4ac86b66-096a-4384-a182-c379dbb54b5d-kube-api-access-5kxgf\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.254777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.254746    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.256619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.256584    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.278589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.278557    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxgf\" (UniqueName: \"kubernetes.io/projected/4ac86b66-096a-4384-a182-c379dbb54b5d-kube-api-access-5kxgf\") pod \"maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.314618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.314277    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:44.533396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.533365    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:44.658327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.658293    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"]
Nov 02 15:37:44.669821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.669789    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d35a86-wtvzd"]
Nov 02 15:37:44.689747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.689715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:44.797885 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:37:44.797788    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac86b66_096a_4384_a182_c379dbb54b5d.slice/crio-ca8c362ee97e7cacdebc9e8490752b5fd0f4665d86a960a694676868a5d2b62a WatchSource:0}: Error finding container ca8c362ee97e7cacdebc9e8490752b5fd0f4665d86a960a694676868a5d2b62a: Status 404 returned error can't find the container with id ca8c362ee97e7cacdebc9e8490752b5fd0f4665d86a960a694676868a5d2b62a
Nov 02 15:37:44.798181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.798151    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"]
Nov 02 15:37:44.877851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:44.877816    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:45.147395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:45.147359    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:45.687890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:45.687848    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6" event={"ID":"4ac86b66-096a-4384-a182-c379dbb54b5d","Type":"ContainerStarted","Data":"0784eb140445bbac4227c2b43b7d726f27c3ab96ae53b9068b46acf279d0d78d"}
Nov 02 15:37:45.688071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:45.687919    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6" event={"ID":"4ac86b66-096a-4384-a182-c379dbb54b5d","Type":"ContainerStarted","Data":"ca8c362ee97e7cacdebc9e8490752b5fd0f4665d86a960a694676868a5d2b62a"}
Nov 02 15:37:45.735237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:45.735188    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6" podStartSLOduration=2.735169571 podStartE2EDuration="2.735169571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:45.731805808 +0000 UTC m=+2459.985759433" watchObservedRunningTime="2026-11-02 15:37:45.735169571 +0000 UTC m=+2459.989123194"
Nov 02 15:37:45.869549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:45.869511    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340" path="/var/lib/kubelet/pods/48e4fcd4-79fb-4511-ba0a-3c3a6e6f8340/volumes"
Nov 02 15:37:46.346968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:46.346927    7019 patch_prober.go:29] interesting pod/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:37:46.347361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:46.346984    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:37:46.521640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:46.521601    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" podUID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:46.584020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:46.583975    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:46.698095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:46.698058    7019 generic.go:356] "Generic (PLEG): container finished" podID="4ac86b66-096a-4384-a182-c379dbb54b5d" containerID="0784eb140445bbac4227c2b43b7d726f27c3ab96ae53b9068b46acf279d0d78d" exitCode=0
Nov 02 15:37:46.698286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:46.698150    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6" event={"ID":"4ac86b66-096a-4384-a182-c379dbb54b5d","Type":"ContainerDied","Data":"0784eb140445bbac4227c2b43b7d726f27c3ab96ae53b9068b46acf279d0d78d"}
Nov 02 15:37:47.654225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:47.654188    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:47.894419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:47.894042    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:48.017256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.017221    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:48.126270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.126226    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-openshift-ca\") pod \"4ac86b66-096a-4384-a182-c379dbb54b5d\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") "
Nov 02 15:37:48.126466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.126384    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-xks-ca\") pod \"4ac86b66-096a-4384-a182-c379dbb54b5d\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") "
Nov 02 15:37:48.126466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.126418    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac86b66-096a-4384-a182-c379dbb54b5d-kube-api-access-5kxgf\" (UniqueName: \"kubernetes.io/projected/4ac86b66-096a-4384-a182-c379dbb54b5d-kube-api-access-5kxgf\") pod \"4ac86b66-096a-4384-a182-c379dbb54b5d\" (UID: \"4ac86b66-096a-4384-a182-c379dbb54b5d\") "
Nov 02 15:37:48.126734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.126700    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-openshift-ca" pod "4ac86b66-096a-4384-a182-c379dbb54b5d" (UID: "4ac86b66-096a-4384-a182-c379dbb54b5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:48.127520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.127492    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.128877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.128854    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac86b66-096a-4384-a182-c379dbb54b5d-kube-api-access-5kxgf" pod "4ac86b66-096a-4384-a182-c379dbb54b5d" (UID: "4ac86b66-096a-4384-a182-c379dbb54b5d"). InnerVolumeSpecName "kube-api-access-5kxgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:48.129034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.129015    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-xks-ca" pod "4ac86b66-096a-4384-a182-c379dbb54b5d" (UID: "4ac86b66-096a-4384-a182-c379dbb54b5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:48.228745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.228711    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac86b66-096a-4384-a182-c379dbb54b5d-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.228745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.228746    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxgf\" (UniqueName: \"kubernetes.io/projected/4ac86b66-096a-4384-a182-c379dbb54b5d-kube-api-access-5kxgf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:48.512235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.512193    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:48.622306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.622258    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" podUID="143c471f-15dc-47a0-b7d3-f10557a08f0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:37:48.719323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.719274    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6" event={"ID":"4ac86b66-096a-4384-a182-c379dbb54b5d","Type":"ContainerDied","Data":"ca8c362ee97e7cacdebc9e8490752b5fd0f4665d86a960a694676868a5d2b62a"}
Nov 02 15:37:48.719817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.719449    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"
Nov 02 15:37:48.719949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.719837    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca8c362ee97e7cacdebc9e8490752b5fd0f4665d86a960a694676868a5d2b62a"
Nov 02 15:37:48.943886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.943858    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:37:48.951346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.951321    7019 scope.go:122] "RemoveContainer" containerID="8092a173dff4a481b4f2241570407a5b5c2116e14ee693192d3fc48b9b997060"
Nov 02 15:37:48.961798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.961776    7019 scope.go:122] "RemoveContainer" containerID="ec894b606e4a1bae5501c4d6a221860d180a96850dfc6f57cc500f712b952b89"
Nov 02 15:37:48.972632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.972607    7019 scope.go:122] "RemoveContainer" containerID="6714f2e9851c84253a49bcc7e46a956fc1bbd09b3a71acf467dc594fb19ada2c"
Nov 02 15:37:48.983312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:48.983280    7019 scope.go:122] "RemoveContainer" containerID="7faaeeed3a4c74c7a409a2e0362957607fc832aa300fee7a914f912bfd78793f"
Nov 02 15:37:49.010213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.008112    7019 scope.go:122] "RemoveContainer" containerID="9fba8a6787ae0f3bfdaa8ed913a5dc0e7afaefe1f3f47c319f8697a8e25f48b0"
Nov 02 15:37:49.022279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.022257    7019 scope.go:122] "RemoveContainer" containerID="e72322e6118744c8ccc4afde82dcf0396def7e8bffb2f2f07f73b3b2c0151b6b"
Nov 02 15:37:49.142231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142206    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-kube-api-access-5rvgr\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-kube-api-access-5rvgr\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142288    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-podinfo\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142331    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-socket\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142354    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istiod-ca-cert\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142385    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-token\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-token\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142410    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-data\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142440    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-certs\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142569    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-credential-socket\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142609    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-envoy\") pod \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\" (UID: \"d8db75dc-11e7-4eeb-9c6f-f3128863a73a\") "
Nov 02 15:37:49.142790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142620    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-socket" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:49.142897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142875    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-credential-socket" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:49.142960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142887    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istiod-ca-cert" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:49.142960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142890    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-certs" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:49.142960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.142930    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-data" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:49.143484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.143458    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.143571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.143490    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.143571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.143505    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.143571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.143519    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.143571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.143532    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.144727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.144701    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-token" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:49.144933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.144908    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-podinfo" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:49.144933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.144924    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-envoy" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:49.145061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.144995    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-kube-api-access-5rvgr" pod "d8db75dc-11e7-4eeb-9c6f-f3128863a73a" (UID: "d8db75dc-11e7-4eeb-9c6f-f3128863a73a"). InnerVolumeSpecName "kube-api-access-5rvgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:49.172465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.172428    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:49.245313 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.245281    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.245477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.245308    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvgr\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-kube-api-access-5rvgr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.245477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.245334    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.245477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.245344    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8db75dc-11e7-4eeb-9c6f-f3128863a73a-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:49.726878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.726846    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" event={"ID":"d8db75dc-11e7-4eeb-9c6f-f3128863a73a","Type":"ContainerDied","Data":"9fba8a6787ae0f3bfdaa8ed913a5dc0e7afaefe1f3f47c319f8697a8e25f48b0"}
Nov 02 15:37:49.727287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.726929    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m" event={"ID":"d8db75dc-11e7-4eeb-9c6f-f3128863a73a","Type":"ContainerDied","Data":"2a54a5ab525511722c62e72589ae15bb76764c99a418292c3f066795cf07b291"}
Nov 02 15:37:49.727287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.727000    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"
Nov 02 15:37:49.828199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.828163    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"]
Nov 02 15:37:49.834733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.834705    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-69ff88-openshift-default-866dfd5ff7-gwt4m"]
Nov 02 15:37:49.864231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:49.864203    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" path="/var/lib/kubelet/pods/d8db75dc-11e7-4eeb-9c6f-f3128863a73a/volumes"
Nov 02 15:37:50.152369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:50.152332    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:51.451677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:51.451607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:51.626780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:51.626745    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:51.756214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:51.756122    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"]
Nov 02 15:37:51.756458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:51.756386    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy" containerID="cri-o://048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d" gracePeriod=30
Nov 02 15:37:51.776374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:51.776328    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:51.947125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:51.947040    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:52.272754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.272697    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:37:52.295280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.295249    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c45682a-e9c5-43f0-8fe0-002daced8c0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c45682a-e9c5-43f0-8fe0-002daced8c0e-plugins-config-volume\") pod \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") "
Nov 02 15:37:52.295736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.295709    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c45682a-e9c5-43f0-8fe0-002daced8c0e-plugins-config-volume" pod "2c45682a-e9c5-43f0-8fe0-002daced8c0e" (UID: "2c45682a-e9c5-43f0-8fe0-002daced8c0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:52.396280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.396247    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c45682a-e9c5-43f0-8fe0-002daced8c0e-kube-api-access-66gf7\" (UniqueName: \"kubernetes.io/projected/2c45682a-e9c5-43f0-8fe0-002daced8c0e-kube-api-access-66gf7\") pod \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\" (UID: \"2c45682a-e9c5-43f0-8fe0-002daced8c0e\") "
Nov 02 15:37:52.396731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.396701    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c45682a-e9c5-43f0-8fe0-002daced8c0e-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:52.398491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.398460    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c45682a-e9c5-43f0-8fe0-002daced8c0e-kube-api-access-66gf7" pod "2c45682a-e9c5-43f0-8fe0-002daced8c0e" (UID: "2c45682a-e9c5-43f0-8fe0-002daced8c0e"). InnerVolumeSpecName "kube-api-access-66gf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:52.497722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.497686    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66gf7\" (UniqueName: \"kubernetes.io/projected/2c45682a-e9c5-43f0-8fe0-002daced8c0e-kube-api-access-66gf7\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:52.645524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.645486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:52.707676 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:52.707626    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:37:52.771323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.771279    7019 generic.go:356] "Generic (PLEG): container finished" podID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" containerID="1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1" exitCode=0
Nov 02 15:37:52.771495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.771342    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"
Nov 02 15:37:52.771495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.771348    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" event={"ID":"2c45682a-e9c5-43f0-8fe0-002daced8c0e","Type":"ContainerDied","Data":"1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1"}
Nov 02 15:37:52.771495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.771467    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7" event={"ID":"2c45682a-e9c5-43f0-8fe0-002daced8c0e","Type":"ContainerDied","Data":"9c52254d7f3a9fcebb1ca00f491b1100073c16b1d96b763ab4c5eecd61ebd6da"}
Nov 02 15:37:52.771495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.771491    7019 scope.go:122] "RemoveContainer" containerID="1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1"
Nov 02 15:37:52.785377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.785355    7019 scope.go:122] "RemoveContainer" containerID="1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1"
Nov 02 15:37:52.785708 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:52.785681    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1\": container with ID starting with 1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1 not found: ID does not exist" containerID="1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1"
Nov 02 15:37:52.785805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.785719    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1"} err="failed to get container status \"1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1\": rpc error: code = NotFound desc = could not find container \"1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1\": container with ID starting with 1c5a629af2cf09b291edab0c47e04fe7bbb3e2f4dbbfe2388983d6c9041314a1 not found: ID does not exist"
Nov 02 15:37:52.827389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.827360    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"]
Nov 02 15:37:52.835610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:52.835583    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-69ff88-7dd9f5c967-9msv7"]
Nov 02 15:37:53.104222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:53.104131    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:53.267671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:53.267073    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:53.410080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:53.410041    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:53.862397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:53.862370    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" path="/var/lib/kubelet/pods/2c45682a-e9c5-43f0-8fe0-002daced8c0e/volumes"
Nov 02 15:37:54.592404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:54.592266    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"]
Nov 02 15:37:54.637871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:54.637836    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"]
Nov 02 15:37:54.778484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:54.778449    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:54.934732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:54.934692    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:55.119991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:55.119951    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:56.202035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.202008    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:37:56.278154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.278115    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143c471f-15dc-47a0-b7d3-f10557a08f0a-kube-api-access-jkcxt\" (UniqueName: \"kubernetes.io/projected/143c471f-15dc-47a0-b7d3-f10557a08f0a-kube-api-access-jkcxt\") pod \"143c471f-15dc-47a0-b7d3-f10557a08f0a\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") "
Nov 02 15:37:56.278331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.278259    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143c471f-15dc-47a0-b7d3-f10557a08f0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c471f-15dc-47a0-b7d3-f10557a08f0a-plugins-config-volume\") pod \"143c471f-15dc-47a0-b7d3-f10557a08f0a\" (UID: \"143c471f-15dc-47a0-b7d3-f10557a08f0a\") "
Nov 02 15:37:56.278955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.278901    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143c471f-15dc-47a0-b7d3-f10557a08f0a-plugins-config-volume" pod "143c471f-15dc-47a0-b7d3-f10557a08f0a" (UID: "143c471f-15dc-47a0-b7d3-f10557a08f0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:56.280992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.280970    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143c471f-15dc-47a0-b7d3-f10557a08f0a-kube-api-access-jkcxt" pod "143c471f-15dc-47a0-b7d3-f10557a08f0a" (UID: "143c471f-15dc-47a0-b7d3-f10557a08f0a"). InnerVolumeSpecName "kube-api-access-jkcxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:56.380104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.380061    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcxt\" (UniqueName: \"kubernetes.io/projected/143c471f-15dc-47a0-b7d3-f10557a08f0a-kube-api-access-jkcxt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:56.380104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.380094    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c471f-15dc-47a0-b7d3-f10557a08f0a-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:56.500834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.500705    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"]
Nov 02 15:37:56.501020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.500918    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7" podUID="495f0861-49cf-49fc-970e-f186501d48f8" containerName="maas-api" containerID="cri-o://1676d5af4b22179b5637df9eb5567d59810effc2ff9da82dec31aad8eafc58af" gracePeriod=30
Nov 02 15:37:56.602516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.602479    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:56.846974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.834158    7019 generic.go:356] "Generic (PLEG): container finished" podID="143c471f-15dc-47a0-b7d3-f10557a08f0a" containerID="37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91" exitCode=0
Nov 02 15:37:56.846974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.834257    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" event={"ID":"143c471f-15dc-47a0-b7d3-f10557a08f0a","Type":"ContainerDied","Data":"37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91"}
Nov 02 15:37:56.846974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.834308    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75" event={"ID":"143c471f-15dc-47a0-b7d3-f10557a08f0a","Type":"ContainerDied","Data":"a6fbbc1d4be93666fd6c166a6f27928f5ad440d9ec4d55cbd50f57223d1bb487"}
Nov 02 15:37:56.846974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.834337    7019 scope.go:122] "RemoveContainer" containerID="37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91"
Nov 02 15:37:56.846974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.834480    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"
Nov 02 15:37:56.847564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.847220    7019 generic.go:356] "Generic (PLEG): container finished" podID="495f0861-49cf-49fc-970e-f186501d48f8" containerID="1676d5af4b22179b5637df9eb5567d59810effc2ff9da82dec31aad8eafc58af" exitCode=0
Nov 02 15:37:56.847564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.847508    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7" event={"ID":"495f0861-49cf-49fc-970e-f186501d48f8","Type":"ContainerDied","Data":"1676d5af4b22179b5637df9eb5567d59810effc2ff9da82dec31aad8eafc58af"}
Nov 02 15:37:56.903522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.903278    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:56.907421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.907396    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:37:56.907791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.907635    7019 scope.go:122] "RemoveContainer" containerID="37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91"
Nov 02 15:37:56.908068 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:56.907978    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91\": container with ID starting with 37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91 not found: ID does not exist" containerID="37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91"
Nov 02 15:37:56.908068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.908012    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91"} err="failed to get container status \"37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91\": rpc error: code = NotFound desc = could not find container \"37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91\": container with ID starting with 37ce554134c5b5f8b3dccc36cbca4da4c5b2b026de88c00d3418d2168711cf91 not found: ID does not exist"
Nov 02 15:37:56.937334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.936385    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"]
Nov 02 15:37:56.941456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.941431    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-31c816-68f57857dc-fnd75"]
Nov 02 15:37:57.004994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.997915    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495f0861-49cf-49fc-970e-f186501d48f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495f0861-49cf-49fc-970e-f186501d48f8-openshift-service-ca\") pod \"495f0861-49cf-49fc-970e-f186501d48f8\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") "
Nov 02 15:37:57.004994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.998056    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-metrics-certs\") pod \"495f0861-49cf-49fc-970e-f186501d48f8\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") "
Nov 02 15:37:57.004994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.998183    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-maas-api-tls\") pod \"495f0861-49cf-49fc-970e-f186501d48f8\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") "
Nov 02 15:37:57.004994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.998293    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495f0861-49cf-49fc-970e-f186501d48f8-kube-api-access-jtb5k\" (UniqueName: \"kubernetes.io/projected/495f0861-49cf-49fc-970e-f186501d48f8-kube-api-access-jtb5k\") pod \"495f0861-49cf-49fc-970e-f186501d48f8\" (UID: \"495f0861-49cf-49fc-970e-f186501d48f8\") "
Nov 02 15:37:57.004994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:56.999032    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495f0861-49cf-49fc-970e-f186501d48f8-openshift-service-ca" pod "495f0861-49cf-49fc-970e-f186501d48f8" (UID: "495f0861-49cf-49fc-970e-f186501d48f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:57.004994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.000241    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/495f0861-49cf-49fc-970e-f186501d48f8-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.017951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.017906    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-maas-api-tls" pod "495f0861-49cf-49fc-970e-f186501d48f8" (UID: "495f0861-49cf-49fc-970e-f186501d48f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:57.019332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.019284    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495f0861-49cf-49fc-970e-f186501d48f8-kube-api-access-jtb5k" pod "495f0861-49cf-49fc-970e-f186501d48f8" (UID: "495f0861-49cf-49fc-970e-f186501d48f8"). InnerVolumeSpecName "kube-api-access-jtb5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:57.022794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.022764    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-metrics-certs" pod "495f0861-49cf-49fc-970e-f186501d48f8" (UID: "495f0861-49cf-49fc-970e-f186501d48f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:37:57.102054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.101963    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.102054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.102003    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtb5k\" (UniqueName: \"kubernetes.io/projected/495f0861-49cf-49fc-970e-f186501d48f8-kube-api-access-jtb5k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.102054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.102018    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/495f0861-49cf-49fc-970e-f186501d48f8-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.181672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.181263    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:57.226669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.225859    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:37:57.307414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307377    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-istio-token\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-istio-token\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307505    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-certs\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307542    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee941490-62a1-40fc-92f9-403df334d9bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee941490-62a1-40fc-92f9-403df334d9bf-istio-podinfo\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307584    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-kube-api-access-cfqnf\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-kube-api-access-cfqnf\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307623    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-socket\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307702    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-data\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307808    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-credential-socket\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307906    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee941490-62a1-40fc-92f9-403df334d9bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee941490-62a1-40fc-92f9-403df334d9bf-istiod-ca-cert\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307951    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-envoy\") pod \"ee941490-62a1-40fc-92f9-403df334d9bf\" (UID: \"ee941490-62a1-40fc-92f9-403df334d9bf\") "
Nov 02 15:37:57.307988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.307967    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-certs" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:57.308530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.308311    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-socket" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:57.309014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.308797    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.309014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.308823    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.309014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.308896    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-data" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:57.309627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.309379    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee941490-62a1-40fc-92f9-403df334d9bf-istiod-ca-cert" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:57.309836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.309799    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-credential-socket" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:57.311516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.311477    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-istio-token" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:57.312235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.312203    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-envoy" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:37:57.312452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.312420    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-kube-api-access-cfqnf" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "kube-api-access-cfqnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:37:57.313245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.313217    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee941490-62a1-40fc-92f9-403df334d9bf-istio-podinfo" pod "ee941490-62a1-40fc-92f9-403df334d9bf" (UID: "ee941490-62a1-40fc-92f9-403df334d9bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:37:57.391932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.391897    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:57.410447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.410414    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.410447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.410447    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.410630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.410457    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee941490-62a1-40fc-92f9-403df334d9bf-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.410630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.410467    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee941490-62a1-40fc-92f9-403df334d9bf-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.410630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.410475    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.410630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.410483    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee941490-62a1-40fc-92f9-403df334d9bf-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.410630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.410490    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfqnf\" (UniqueName: \"kubernetes.io/projected/ee941490-62a1-40fc-92f9-403df334d9bf-kube-api-access-cfqnf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:57.449907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.449876    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"]
Nov 02 15:37:57.491410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.490096    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-31c816-8g8nc"]
Nov 02 15:37:57.553588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.553552    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:57.856776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.856686    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"
Nov 02 15:37:57.858774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.858744    7019 generic.go:356] "Generic (PLEG): container finished" podID="ee941490-62a1-40fc-92f9-403df334d9bf" containerID="048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d" exitCode=0
Nov 02 15:37:57.858898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.858809    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"
Nov 02 15:37:57.863022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.862994    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143c471f-15dc-47a0-b7d3-f10557a08f0a" path="/var/lib/kubelet/pods/143c471f-15dc-47a0-b7d3-f10557a08f0a/volumes"
Nov 02 15:37:57.863542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.863524    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55adfe75-2cdb-4356-8dde-212a00ac539e" path="/var/lib/kubelet/pods/55adfe75-2cdb-4356-8dde-212a00ac539e/volumes"
Nov 02 15:37:57.864150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.864130    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7" event={"ID":"495f0861-49cf-49fc-970e-f186501d48f8","Type":"ContainerDied","Data":"df7623489cfa7da488709d00d4eb1c26b23599648cd34a46e3f866ee48a413e2"}
Nov 02 15:37:57.864267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.864167    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" event={"ID":"ee941490-62a1-40fc-92f9-403df334d9bf","Type":"ContainerDied","Data":"048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d"}
Nov 02 15:37:57.864267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.864191    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8" event={"ID":"ee941490-62a1-40fc-92f9-403df334d9bf","Type":"ContainerDied","Data":"aad775ecdcf0c18303fc73b6e88a9bd5d3bfb7112722bd3a357d9358f805cd31"}
Nov 02 15:37:57.864267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.864223    7019 scope.go:122] "RemoveContainer" containerID="1676d5af4b22179b5637df9eb5567d59810effc2ff9da82dec31aad8eafc58af"
Nov 02 15:37:57.881612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.881589    7019 scope.go:122] "RemoveContainer" containerID="048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d"
Nov 02 15:37:57.892178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.892157    7019 scope.go:122] "RemoveContainer" containerID="d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20"
Nov 02 15:37:57.901721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.901642    7019 scope.go:122] "RemoveContainer" containerID="048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d"
Nov 02 15:37:57.901980 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:57.901961    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d\": container with ID starting with 048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d not found: ID does not exist" containerID="048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d"
Nov 02 15:37:57.902034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.901987    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d"} err="failed to get container status \"048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d\": rpc error: code = NotFound desc = could not find container \"048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d\": container with ID starting with 048f6c8ecd1379375758d5a1e469bfe6366e787cc6be47da9afb373e0ced6d8d not found: ID does not exist"
Nov 02 15:37:57.902034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.902007    7019 scope.go:122] "RemoveContainer" containerID="d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20"
Nov 02 15:37:57.902304 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:37:57.902282    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20\": container with ID starting with d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20 not found: ID does not exist" containerID="d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20"
Nov 02 15:37:57.902409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.902310    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20"} err="failed to get container status \"d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20\": rpc error: code = NotFound desc = could not find container \"d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20\": container with ID starting with d2c8ae69d910deee50eefd39591ad9928270bcaf23cc47769fa770f7739c4e20 not found: ID does not exist"
Nov 02 15:37:57.920988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.920959    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"]
Nov 02 15:37:57.924022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.924000    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-31c816-7b468cfb67-88bl7"]
Nov 02 15:37:57.997759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:57.997719    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"]
Nov 02 15:37:58.002363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:58.002329    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d35a86-openshift-default-75f7677c87-7xgx8"]
Nov 02 15:37:58.030279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:58.030248    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:58.747200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:58.747139    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:58.902605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:58.902549    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:59.436476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.436432    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:37:59.593727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.593678    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq" podUID="dde78ef6-9d08-402d-9fb4-5fa3a2b50074" containerName="payload-processing" containerID="cri-o://7ff97cdf44b7ce87487fb1bd548f293fa14c56e73c1985b9e4885b7c8460cd78" gracePeriod=25
Nov 02 15:37:59.638733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.638696    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" podUID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" containerName="payload-pre-processing" containerID="cri-o://dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a" gracePeriod=25
Nov 02 15:37:59.863463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.863435    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495f0861-49cf-49fc-970e-f186501d48f8" path="/var/lib/kubelet/pods/495f0861-49cf-49fc-970e-f186501d48f8/volumes"
Nov 02 15:37:59.864053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.864002    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" path="/var/lib/kubelet/pods/ee941490-62a1-40fc-92f9-403df334d9bf/volumes"
Nov 02 15:37:59.878816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.878792    7019 generic.go:356] "Generic (PLEG): container finished" podID="dde78ef6-9d08-402d-9fb4-5fa3a2b50074" containerID="7ff97cdf44b7ce87487fb1bd548f293fa14c56e73c1985b9e4885b7c8460cd78" exitCode=0
Nov 02 15:37:59.878957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.878877    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq" event={"ID":"dde78ef6-9d08-402d-9fb4-5fa3a2b50074","Type":"ContainerDied","Data":"7ff97cdf44b7ce87487fb1bd548f293fa14c56e73c1985b9e4885b7c8460cd78"}
Nov 02 15:37:59.878957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.878943    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq" event={"ID":"dde78ef6-9d08-402d-9fb4-5fa3a2b50074","Type":"ContainerDied","Data":"0c0d3da29427aef16a326847f7b0c61431a61364801b971337dc97b4e45d235e"}
Nov 02 15:37:59.879034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.878966    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c0d3da29427aef16a326847f7b0c61431a61364801b971337dc97b4e45d235e"
Nov 02 15:37:59.888379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.888356    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:37:59.948572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.948538    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-plugins-config-volume\") pod \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") "
Nov 02 15:37:59.948797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.948765    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-kube-api-access-xkfwj\" (UniqueName: \"kubernetes.io/projected/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-kube-api-access-xkfwj\") pod \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\" (UID: \"dde78ef6-9d08-402d-9fb4-5fa3a2b50074\") "
Nov 02 15:37:59.948981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.948952    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-plugins-config-volume" pod "dde78ef6-9d08-402d-9fb4-5fa3a2b50074" (UID: "dde78ef6-9d08-402d-9fb4-5fa3a2b50074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:37:59.949376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.949355    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:37:59.950851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:37:59.950824    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-kube-api-access-xkfwj" pod "dde78ef6-9d08-402d-9fb4-5fa3a2b50074" (UID: "dde78ef6-9d08-402d-9fb4-5fa3a2b50074"). InnerVolumeSpecName "kube-api-access-xkfwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:00.050366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:00.050275    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkfwj\" (UniqueName: \"kubernetes.io/projected/dde78ef6-9d08-402d-9fb4-5fa3a2b50074-kube-api-access-xkfwj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:00.263492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:00.263447    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" podUID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:38:00.886058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:00.886027    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"
Nov 02 15:38:00.943410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:00.943372    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"]
Nov 02 15:38:00.956206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:00.956170    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2e4ebc-bbf5fdccc-q79mq"]
Nov 02 15:38:01.314190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:01.313642    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:01.864619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:01.864583    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dde78ef6-9d08-402d-9fb4-5fa3a2b50074" path="/var/lib/kubelet/pods/dde78ef6-9d08-402d-9fb4-5fa3a2b50074/volumes"
Nov 02 15:38:02.640486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:02.640448    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:03.010705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:03.010602    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:04.456583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:04.456541    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:04.601235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:04.601191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:04.767456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:04.767337    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"]
Nov 02 15:38:04.767890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:04.767860    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" containerID="cri-o://dda6dfc4286d63a17ef4bc7a517faa0ff1991ea43f215922b4abefd95e011287" gracePeriod=30
Nov 02 15:38:04.829352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:04.829311    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:05.365904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:05.365864    7019 patch_prober.go:29] interesting pod/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:38:05.366114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:05.365939    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:38:06.226540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.226465    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"]
Nov 02 15:38:06.227866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227835    7019 cpu_manager.go:414] "containerMap: removing container" podUID="495f0861-49cf-49fc-970e-f186501d48f8" containerName="maas-api"
Nov 02 15:38:06.227866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227866    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495f0861-49cf-49fc-970e-f186501d48f8" containerName="maas-api"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227892    7019 cpu_manager.go:414] "containerMap: removing container" podUID="143c471f-15dc-47a0-b7d3-f10557a08f0a" containerName="payload-pre-processing"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227900    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143c471f-15dc-47a0-b7d3-f10557a08f0a" containerName="payload-pre-processing"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227914    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4ac86b66-096a-4384-a182-c379dbb54b5d" containerName="revoke-keys"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227922    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac86b66-096a-4384-a182-c379dbb54b5d" containerName="revoke-keys"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227974    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dde78ef6-9d08-402d-9fb4-5fa3a2b50074" containerName="payload-processing"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227982    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dde78ef6-9d08-402d-9fb4-5fa3a2b50074" containerName="payload-processing"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.227993    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228000    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228027    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" containerName="payload-processing"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228034    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" containerName="payload-processing"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228043    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228050    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228061    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228068    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy"
Nov 02 15:38:06.228077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228083    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228089    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228231    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228244    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228256    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="495f0861-49cf-49fc-970e-f186501d48f8" containerName="maas-api"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228269    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="143c471f-15dc-47a0-b7d3-f10557a08f0a" containerName="payload-pre-processing"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228279    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac86b66-096a-4384-a182-c379dbb54b5d" containerName="revoke-keys"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228301    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dde78ef6-9d08-402d-9fb4-5fa3a2b50074" containerName="payload-processing"
Nov 02 15:38:06.228952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.228312    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c45682a-e9c5-43f0-8fe0-002daced8c0e" containerName="payload-processing"
Nov 02 15:38:06.233820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.233794    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.259288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.259253    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"]
Nov 02 15:38:06.262894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.262867    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-246137\"/\"default-dockercfg-d6vcj\""
Nov 02 15:38:06.262894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.262872    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-246137\"/\"openshift-service-ca.crt\""
Nov 02 15:38:06.262894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.262882    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-246137\"/\"rate-test-model-246137-kserve-self-signed-certs\""
Nov 02 15:38:06.273018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.272994    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-246137\"/\"kube-root-ca.crt\""
Nov 02 15:38:06.348758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.348715    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-dshm\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.348977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.348888    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmmv\" (UniqueName: \"kubernetes.io/projected/15379f67-4c39-4ab6-accc-ae1fe5dc679b-kube-api-access-swmmv\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.349164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.349141    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tmp-dir\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.349231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.349183    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-model-cache\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.349231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.349210    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-home\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.349354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.349251    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tls-certs\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.356553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.356517    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:06.450671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.450604    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tmp-dir\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.450671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.450687    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-model-cache\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.450958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.450710    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-home\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.450958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.450725    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tls-certs\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.450958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.450759    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-dshm\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.450958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.450846    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmmv\" (UniqueName: \"kubernetes.io/projected/15379f67-4c39-4ab6-accc-ae1fe5dc679b-kube-api-access-swmmv\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.451201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.451081    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tmp-dir\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.451401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.451374    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-home\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.451401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.451393    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-model-cache\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.453840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.453817    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-dshm\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.454147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.454124    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tls-certs\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.475903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.475868    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmmv\" (UniqueName: \"kubernetes.io/projected/15379f67-4c39-4ab6-accc-ae1fe5dc679b-kube-api-access-swmmv\") pod \"rate-test-model-246137-kserve-769cddb97c-xrh49\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") " pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.512083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.511985    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:06.571204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.570916    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:06.957534 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:38:06.957498    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15379f67_4c39_4ab6_accc_ae1fe5dc679b.slice/crio-bdaabfb3d92aaaec1b7c870c56134db99af92a16f15bd5ebbf218b1d41d4c42f WatchSource:0}: Error finding container bdaabfb3d92aaaec1b7c870c56134db99af92a16f15bd5ebbf218b1d41d4c42f: Status 404 returned error can't find the container with id bdaabfb3d92aaaec1b7c870c56134db99af92a16f15bd5ebbf218b1d41d4c42f
Nov 02 15:38:06.958459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:06.958431    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"]
Nov 02 15:38:07.956983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:07.956942    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49" event={"ID":"15379f67-4c39-4ab6-accc-ae1fe5dc679b","Type":"ContainerStarted","Data":"35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23"}
Nov 02 15:38:07.957497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:07.956996    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49" event={"ID":"15379f67-4c39-4ab6-accc-ae1fe5dc679b","Type":"ContainerStarted","Data":"bdaabfb3d92aaaec1b7c870c56134db99af92a16f15bd5ebbf218b1d41d4c42f"}
Nov 02 15:38:08.008280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:08.008218    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49" podStartSLOduration=2.008200562 podStartE2EDuration="2.008200562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:08.004144283 +0000 UTC m=+2482.258097908" watchObservedRunningTime="2026-11-02 15:38:08.008200562 +0000 UTC m=+2482.262154186"
Nov 02 15:38:08.039866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:08.039817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:08.572057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:08.572020    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:09.792757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:09.792723    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:09.976514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:09.976476    7019 generic.go:356] "Generic (PLEG): container finished" podID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerID="dda6dfc4286d63a17ef4bc7a517faa0ff1991ea43f215922b4abefd95e011287" exitCode=0
Nov 02 15:38:09.976710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:09.976549    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" event={"ID":"ab9b3c83-b754-4bbe-94fa-8036234b0e65","Type":"ContainerDied","Data":"dda6dfc4286d63a17ef4bc7a517faa0ff1991ea43f215922b4abefd95e011287"}
Nov 02 15:38:09.976710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:09.976625    7019 scope.go:122] "RemoveContainer" containerID="89deddf99ecfba4fdcef094eff85129c2a1abd5d4be6a7f6b41c010a987b1d83"
Nov 02 15:38:10.060530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.060502    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:38:10.075371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.075344    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:10.207963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.207932    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-token\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-token\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.207963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.207967    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-kube-api-access-dqhn6\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-kube-api-access-dqhn6\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208006    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-data\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208026    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-credential-socket\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208058    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-socket\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208088    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-certs\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208173    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istiod-ca-cert\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208324    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-podinfo\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208437    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-data" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:10.208614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208484    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-socket" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:10.208614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208509    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-certs" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:10.208614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208535    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-envoy\") pod \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\" (UID: \"ab9b3c83-b754-4bbe-94fa-8036234b0e65\") "
Nov 02 15:38:10.208797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208687    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-credential-socket" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:10.208797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.208718    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istiod-ca-cert" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:38:10.209220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.209193    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.209220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.209220    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.209421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.209233    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.209421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.209249    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.209421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.209263    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.210331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.210309    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-kube-api-access-dqhn6" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "kube-api-access-dqhn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:10.210476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.210459    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-token" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:10.210574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.210555    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-podinfo" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:38:10.210749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.210732    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-envoy" pod "ab9b3c83-b754-4bbe-94fa-8036234b0e65" (UID: "ab9b3c83-b754-4bbe-94fa-8036234b0e65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:10.263803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.263752    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" podUID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:38:10.310278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.310207    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.310278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.310241    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqhn6\" (UniqueName: \"kubernetes.io/projected/ab9b3c83-b754-4bbe-94fa-8036234b0e65-kube-api-access-dqhn6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.310278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.310259    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.310278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.310272    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab9b3c83-b754-4bbe-94fa-8036234b0e65-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:10.986215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.986121    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"
Nov 02 15:38:10.986215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.986170    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j" event={"ID":"ab9b3c83-b754-4bbe-94fa-8036234b0e65","Type":"ContainerDied","Data":"0e35bd228b432cef48795c748ef67922f1ebe4f5c396ce741b084ff9df66db58"}
Nov 02 15:38:10.986846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:10.986240    7019 scope.go:122] "RemoveContainer" containerID="dda6dfc4286d63a17ef4bc7a517faa0ff1991ea43f215922b4abefd95e011287"
Nov 02 15:38:11.056793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:11.056753    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"]
Nov 02 15:38:11.060344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:11.060322    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-31c816-openshift-default-7f485f86fc-pb52j"]
Nov 02 15:38:11.542538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:11.542474    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:11.862973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:11.862941    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" path="/var/lib/kubelet/pods/ab9b3c83-b754-4bbe-94fa-8036234b0e65/volumes"
Nov 02 15:38:13.152178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:13.148679    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:14.850798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:14.850757    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:15.398333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:15.398294    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:15.523384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:15.523352    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"]
Nov 02 15:38:15.535587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:15.535558    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2e4ebc-2xqm6"]
Nov 02 15:38:15.574039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:15.574005    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:15.726585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:15.726488    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:15.863110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:15.863078    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac86b66-096a-4384-a182-c379dbb54b5d" path="/var/lib/kubelet/pods/4ac86b66-096a-4384-a182-c379dbb54b5d/volumes"
Nov 02 15:38:16.035879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:16.035771    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:16.572257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:16.572218    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:16.591414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:16.591383    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:17.059810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:17.059774    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:38:17.379791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:17.379753    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:18.991626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:18.991587    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:19.910474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:19.910438    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:38:19.938142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:19.938116    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-plugins-config-volume\") pod \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") "
Nov 02 15:38:19.938301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:19.938214    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-kube-api-access-m4bzx\" (UniqueName: \"kubernetes.io/projected/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-kube-api-access-m4bzx\") pod \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\" (UID: \"d21df4f0-77ae-4f95-a5bc-0716effbdd3e\") "
Nov 02 15:38:19.938476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:19.938453    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-plugins-config-volume" pod "d21df4f0-77ae-4f95-a5bc-0716effbdd3e" (UID: "d21df4f0-77ae-4f95-a5bc-0716effbdd3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:38:19.938676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:19.938660    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:19.940308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:19.940288    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-kube-api-access-m4bzx" pod "d21df4f0-77ae-4f95-a5bc-0716effbdd3e" (UID: "d21df4f0-77ae-4f95-a5bc-0716effbdd3e"). InnerVolumeSpecName "kube-api-access-m4bzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:20.040225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.040194    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bzx\" (UniqueName: \"kubernetes.io/projected/d21df4f0-77ae-4f95-a5bc-0716effbdd3e-kube-api-access-m4bzx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:20.076806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.076770    7019 generic.go:356] "Generic (PLEG): container finished" podID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" containerID="dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a" exitCode=0
Nov 02 15:38:20.076979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.076825    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"
Nov 02 15:38:20.076979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.076867    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" event={"ID":"d21df4f0-77ae-4f95-a5bc-0716effbdd3e","Type":"ContainerDied","Data":"dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a"}
Nov 02 15:38:20.076979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.076938    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6" event={"ID":"d21df4f0-77ae-4f95-a5bc-0716effbdd3e","Type":"ContainerDied","Data":"0e64b631e338ea3fe2f616424fa248b203b3a13aa3181c194434d2bf6e0baec6"}
Nov 02 15:38:20.076979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.076970    7019 scope.go:122] "RemoveContainer" containerID="dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a"
Nov 02 15:38:20.089621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.089604    7019 scope.go:122] "RemoveContainer" containerID="dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a"
Nov 02 15:38:20.089940 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:38:20.089921    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a\": container with ID starting with dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a not found: ID does not exist" containerID="dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a"
Nov 02 15:38:20.089999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.089951    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a"} err="failed to get container status \"dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a\": rpc error: code = NotFound desc = could not find container \"dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a\": container with ID starting with dc25d034d9e8129ebf5f67330b84ada8d43d4f3e0789252e389ed8dcd12ca85a not found: ID does not exist"
Nov 02 15:38:20.136118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.136086    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"]
Nov 02 15:38:20.140422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.140393    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2e4ebc-55477db7ff-2qtj6"]
Nov 02 15:38:20.604368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:20.604327    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:21.863968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:21.863933    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" path="/var/lib/kubelet/pods/d21df4f0-77ae-4f95-a5bc-0716effbdd3e/volumes"
Nov 02 15:38:22.215533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:22.215423    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:22.707158 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:38:22.707118    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:38:22.746635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:22.746589    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:22.936516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:22.936479    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:23.025063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:23.024967    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"]
Nov 02 15:38:23.025237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:23.025170    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy" containerID="cri-o://cf6f6954b12a933dcc30ccba3a8630386de1a57a9f0869fd0898a46ac0492821" gracePeriod=30
Nov 02 15:38:23.109129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:23.109082    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:23.906245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:23.906208    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:24.053681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:24.053633    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:24.413038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:24.413008    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:24.588582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:24.588546    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:25.631236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:25.631191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:27.276686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:27.276637    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:38:27.485812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:27.485248    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:28.151421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.151390    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:28.158814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.158786    7019 generic.go:356] "Generic (PLEG): container finished" podID="29329576-824e-44bb-a825-b93ab7391426" containerID="cf6f6954b12a933dcc30ccba3a8630386de1a57a9f0869fd0898a46ac0492821" exitCode=0
Nov 02 15:38:28.158931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.158855    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" event={"ID":"29329576-824e-44bb-a825-b93ab7391426","Type":"ContainerDied","Data":"cf6f6954b12a933dcc30ccba3a8630386de1a57a9f0869fd0898a46ac0492821"}
Nov 02 15:38:28.158931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.158917    7019 scope.go:122] "RemoveContainer" containerID="592225f8325dfea14ba8db1eced35d4fe12af35bbfd52a0b0db67c4cd5ee542f"
Nov 02 15:38:28.310151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.310085    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:38:28.334131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.334096    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:28.409017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.408975    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-certs\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409038    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29329576-824e-44bb-a825-b93ab7391426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29329576-824e-44bb-a825-b93ab7391426-istio-podinfo\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409098    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-kube-api-access-tjsqr\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-kube-api-access-tjsqr\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409192    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-credential-socket\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409227    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-envoy\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409283    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-socket\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409295    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-certs" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:28.409479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409376    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29329576-824e-44bb-a825-b93ab7391426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29329576-824e-44bb-a825-b93ab7391426-istiod-ca-cert\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409419    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-istio-token\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-istio-token\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.409729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409488    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-data\") pod \"29329576-824e-44bb-a825-b93ab7391426\" (UID: \"29329576-824e-44bb-a825-b93ab7391426\") "
Nov 02 15:38:28.410540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.409985    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-socket" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:28.410540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.410098    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.410540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.410118    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.410540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.410185    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-credential-socket" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:28.410540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.410505    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29329576-824e-44bb-a825-b93ab7391426-istiod-ca-cert" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:38:28.410962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.410578    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-data" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:28.413054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.413027    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-kube-api-access-tjsqr" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "kube-api-access-tjsqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:28.413739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.413004    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29329576-824e-44bb-a825-b93ab7391426-istio-podinfo" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:38:28.414078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.414046    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-istio-token" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:38:28.414562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.414532    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-envoy" pod "29329576-824e-44bb-a825-b93ab7391426" (UID: "29329576-824e-44bb-a825-b93ab7391426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:38:28.511821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.511778    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29329576-824e-44bb-a825-b93ab7391426-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.511821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.511813    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.511821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.511825    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.512071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.511837    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29329576-824e-44bb-a825-b93ab7391426-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.512071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.511849    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsqr\" (UniqueName: \"kubernetes.io/projected/29329576-824e-44bb-a825-b93ab7391426-kube-api-access-tjsqr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.512071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.511863    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:28.512071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:28.511874    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29329576-824e-44bb-a825-b93ab7391426-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:38:29.174956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.174915    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv" event={"ID":"29329576-824e-44bb-a825-b93ab7391426","Type":"ContainerDied","Data":"a043574f70a187c5a79a10fbf4b860579db6c7b6f23e294f3ae8edbf226709ed"}
Nov 02 15:38:29.175196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.174957    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"
Nov 02 15:38:29.175420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.175193    7019 scope.go:122] "RemoveContainer" containerID="cf6f6954b12a933dcc30ccba3a8630386de1a57a9f0869fd0898a46ac0492821"
Nov 02 15:38:29.244502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.244463    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"]
Nov 02 15:38:29.257352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.257315    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2e4ebc-openshift-default-bd4fb5447-fptlv"]
Nov 02 15:38:29.322717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.322676    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"]
Nov 02 15:38:29.323303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323279    7019 cpu_manager.go:414] "containerMap: removing container" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy"
Nov 02 15:38:29.323364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323302    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy"
Nov 02 15:38:29.323364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323331    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy"
Nov 02 15:38:29.323364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323336    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy"
Nov 02 15:38:29.323364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323351    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" containerName="payload-pre-processing"
Nov 02 15:38:29.323364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323358    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" containerName="payload-pre-processing"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323371    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323376    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323481    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21df4f0-77ae-4f95-a5bc-0716effbdd3e" containerName="payload-pre-processing"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323489    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323496    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323509    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323516    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8db75dc-11e7-4eeb-9c6f-f3128863a73a" containerName="istio-proxy"
Nov 02 15:38:29.323535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.323524    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee941490-62a1-40fc-92f9-403df334d9bf" containerName="istio-proxy"
Nov 02 15:38:29.328440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.328423    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.358770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.358739    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"]
Nov 02 15:38:29.396592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.396561    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-6f2e0c-serving-cert\""
Nov 02 15:38:29.425680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.425592    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj7z7\" (UniqueName: \"kubernetes.io/projected/db681668-d7cc-4da3-8835-1b72ef6687f1-kube-api-access-gj7z7\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.425840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.425690    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.425840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.425800    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.425933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.425855    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db681668-d7cc-4da3-8835-1b72ef6687f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.470326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.470279    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:29.529097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.528221    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.529097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.528311    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db681668-d7cc-4da3-8835-1b72ef6687f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.529097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.528492    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj7z7\" (UniqueName: \"kubernetes.io/projected/db681668-d7cc-4da3-8835-1b72ef6687f1-kube-api-access-gj7z7\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.529097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.528585    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.531083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.531054    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db681668-d7cc-4da3-8835-1b72ef6687f1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.536220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.536191    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.538803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.538779    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.557512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.557471    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj7z7\" (UniqueName: \"kubernetes.io/projected/db681668-d7cc-4da3-8835-1b72ef6687f1-kube-api-access-gj7z7\") pod \"maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.643463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.643431    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:29.669188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.669158    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:29.840989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.840945    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:29.870485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.870451    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29329576-824e-44bb-a825-b93ab7391426" path="/var/lib/kubelet/pods/29329576-824e-44bb-a825-b93ab7391426/volumes"
Nov 02 15:38:29.951142 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:38:29.951113    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb681668_d7cc_4da3_8835_1b72ef6687f1.slice/crio-1fa5f2ea48d80c5d093bea207825fd3294360c5a2e771e1408370e012cb226ae WatchSource:0}: Error finding container 1fa5f2ea48d80c5d093bea207825fd3294360c5a2e771e1408370e012cb226ae: Status 404 returned error can't find the container with id 1fa5f2ea48d80c5d093bea207825fd3294360c5a2e771e1408370e012cb226ae
Nov 02 15:38:29.952031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.952008    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"]
Nov 02 15:38:29.952569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:29.952554    7019 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:38:30.184449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:30.184407    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58" event={"ID":"db681668-d7cc-4da3-8835-1b72ef6687f1","Type":"ContainerStarted","Data":"1fa5f2ea48d80c5d093bea207825fd3294360c5a2e771e1408370e012cb226ae"}
Nov 02 15:38:30.865803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:30.865768    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:31.016078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.016035    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"]
Nov 02 15:38:31.016827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.016803    7019 cpu_manager.go:414] "containerMap: removing container" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy"
Nov 02 15:38:31.016827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.016826    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29329576-824e-44bb-a825-b93ab7391426" containerName="istio-proxy"
Nov 02 15:38:31.017044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.017032    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab9b3c83-b754-4bbe-94fa-8036234b0e65" containerName="istio-proxy"
Nov 02 15:38:31.020314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.020294    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.051718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.051686    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0418a181-53a9-43a2-b261-dac04482277f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.052003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.051974    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56h7\" (UniqueName: \"kubernetes.io/projected/0418a181-53a9-43a2-b261-dac04482277f-kube-api-access-w56h7\") pod \"payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.052628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.052609    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f6f1a0\""
Nov 02 15:38:31.053290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.053265    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"]
Nov 02 15:38:31.062942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.061488    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f6f1a0-dockercfg-7pz4z\""
Nov 02 15:38:31.114328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.114285    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:31.184730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.165542    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0418a181-53a9-43a2-b261-dac04482277f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.184730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.165615    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56h7\" (UniqueName: \"kubernetes.io/projected/0418a181-53a9-43a2-b261-dac04482277f-kube-api-access-w56h7\") pod \"payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.184730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.166912    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0418a181-53a9-43a2-b261-dac04482277f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.258453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.258394    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56h7\" (UniqueName: \"kubernetes.io/projected/0418a181-53a9-43a2-b261-dac04482277f-kube-api-access-w56h7\") pod \"payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.260142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.260083    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58" event={"ID":"db681668-d7cc-4da3-8835-1b72ef6687f1","Type":"ContainerStarted","Data":"232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29"}
Nov 02 15:38:31.261591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.261545    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:31.311353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.311288    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58" podStartSLOduration=1.983661122 podStartE2EDuration="2.311274995s" totalImagesPullingTime="327.613873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:31.309397333 +0000 UTC m=+2505.563350970" watchObservedRunningTime="2026-11-02 15:38:31.311274995 +0000 UTC m=+2505.565228619"
Nov 02 15:38:31.335578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.335548    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:31.634195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.634166    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:31.690394 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:38:31.690367    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0418a181_53a9_43a2_b261_dac04482277f.slice/crio-cc9dc23e184b5efa43c332af44a2df8dcb7a725b7cac38373fd874128611ed25 WatchSource:0}: Error finding container cc9dc23e184b5efa43c332af44a2df8dcb7a725b7cac38373fd874128611ed25: Status 404 returned error can't find the container with id cc9dc23e184b5efa43c332af44a2df8dcb7a725b7cac38373fd874128611ed25
Nov 02 15:38:31.694018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:31.693984    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"]
Nov 02 15:38:32.270612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:32.270560    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" event={"ID":"0418a181-53a9-43a2-b261-dac04482277f","Type":"ContainerStarted","Data":"9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613"}
Nov 02 15:38:32.271134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:32.270638    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" event={"ID":"0418a181-53a9-43a2-b261-dac04482277f","Type":"ContainerStarted","Data":"cc9dc23e184b5efa43c332af44a2df8dcb7a725b7cac38373fd874128611ed25"}
Nov 02 15:38:32.271134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:32.270915    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:33.450751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:33.450057    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:35.125689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:35.125605    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:36.703670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:36.698724    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:38.285070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:38.285036    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:38:38.332855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:38.332803    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" podStartSLOduration=8.332790029 podStartE2EDuration="8.332790029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:32.321882545 +0000 UTC m=+2506.575836171" watchObservedRunningTime="2026-11-02 15:38:38.332790029 +0000 UTC m=+2512.586743697"
Nov 02 15:38:38.395823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:38.395775    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:39.460949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:39.460913    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:40.098857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:40.098780    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:41.791657 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:41.791604    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:43.281837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:43.281804    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:38:43.502669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:43.500486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:45.104803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:45.104766    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:46.790418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:46.790380    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:48.627748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:48.627709    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:49.143639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:49.143610    7019 scope.go:122] "RemoveContainer" containerID="7ff97cdf44b7ce87487fb1bd548f293fa14c56e73c1985b9e4885b7c8460cd78"
Nov 02 15:38:49.156461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:49.156437    7019 scope.go:122] "RemoveContainer" containerID="e30226dfe8d1f639c24dda15bb512d8d7410b37bf663b674e18d29ec1a4d13ce"
Nov 02 15:38:49.167349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:49.167326    7019 scope.go:122] "RemoveContainer" containerID="a6e402a7da5154962965e2b1ef723015717bfec5590810dd7b939af43508ac01"
Nov 02 15:38:49.177623 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:49.177601    7019 scope.go:122] "RemoveContainer" containerID="bc72235027db4a68dfa9fdce9703a0722c7c9470d43bb5a767b7f068215425d6"
Nov 02 15:38:49.188486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:49.188469    7019 scope.go:122] "RemoveContainer" containerID="945ef0c65e242f8c62c45c397ce42bf35da119d7ce0256358b14dc623b12f585"
Nov 02 15:38:49.303944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:49.303905    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:50.090781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:50.090725    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:50.242762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:50.242722    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:51.835358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:51.835293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:52.706556 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:38:52.706518    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:38:53.391993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:53.391959    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:54.997015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:54.996979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:56.788870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:56.788832    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:58.307223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:58.307187    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:38:59.918263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:38:59.918219    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:00.078531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:00.078490    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:00.267685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:00.267567    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:01.601597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:01.601549    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:02.774840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:02.774796    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:03.304282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:03.304247    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:04.875624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:04.875573    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:06.455736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:06.455700    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:08.106221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:08.106185    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:08.332083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:08.332045    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:08.779667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:08.776469    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:09.226594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.226547    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"]
Nov 02 15:39:09.234567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.234514    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.271713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.269772    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-797488-openshift-default-dockercfg-crhd7\""
Nov 02 15:39:09.276268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.276233    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"]
Nov 02 15:39:09.330415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.330370    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:09.394693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.393502    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.394693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.393600    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-credential-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.394693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.393703    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbvq\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-kube-api-access-hnbvq\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.394693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.393748    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-token\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.394693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.393799    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-podinfo\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.394693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.393829    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-data\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.394693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.393855    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istiod-ca-cert\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.403682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.401874    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-certs\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.403682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.402033    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-envoy\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.514694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.514491    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.515372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.515091    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.515625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.515606    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-credential-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.516134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.516034    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-credential-socket\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.516631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.516609    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbvq\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-kube-api-access-hnbvq\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.516823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.516805    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-token\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.517208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.517189    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-podinfo\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.517821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.517801    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-data\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.518224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.518202    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istiod-ca-cert\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.518326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.518274    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-certs\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.518559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.518472    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-envoy\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.519284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.518942    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-data\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.519284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.519214    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-certs\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.519284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.519278    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istiod-ca-cert\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.523525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.523502    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-podinfo\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.525489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.525468    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-envoy\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.540586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.540559    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbvq\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-kube-api-access-hnbvq\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.548363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.548335    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-token\") pod \"e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") " pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.562557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.561218    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:09.600542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.600506    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:09.912100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.910896    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:09.917972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.917917    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"]
Nov 02 15:39:09.937301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.937247    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:09.937489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.937344    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:09.937489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:09.937396    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:10.582536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:10.582492    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" event={"ID":"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf","Type":"ContainerStarted","Data":"d7fd9341859c315bf8a6ff50b0c28a7c84cb373ef80398ef7df0f0a1abc8de41"}
Nov 02 15:39:10.582990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:10.582562    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" event={"ID":"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf","Type":"ContainerStarted","Data":"999a4b4ca693a0bc9a2958ab58510b14839a8ed2bd4f423ee9e5839bbd8643ef"}
Nov 02 15:39:10.633587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:10.633538    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podStartSLOduration=1.633524638 podStartE2EDuration="1.633524638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:10.633164299 +0000 UTC m=+2544.887117933" watchObservedRunningTime="2026-11-02 15:39:10.633524638 +0000 UTC m=+2544.887478261"
Nov 02 15:39:11.571675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:11.570830    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:11.608399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:11.608364    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:11.676584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:11.676550    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:12.618231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:12.618196    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:12.619177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:12.619156    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:39:13.131845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:13.131803    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:13.389306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:13.389027    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:13.642979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:13.642881    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:15.239954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:15.239922    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:15.512811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:15.512198    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:15.703671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:15.694974    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:16.937455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:16.937420    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"]
Nov 02 15:39:16.941971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:16.941944    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:16.961570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:16.961537    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"]
Nov 02 15:39:16.980328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:16.979881    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:39:16.980328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:16.980057    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:39:17.097657 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.097608    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.097870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.097737    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.097951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.097903    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt95m\" (UniqueName: \"kubernetes.io/projected/658e5f59-662f-4f04-8e70-44c375ccba55-kube-api-access-qt95m\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.202127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.202032    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.202415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.202393    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.203230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.203204    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.203697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.203670    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt95m\" (UniqueName: \"kubernetes.io/projected/658e5f59-662f-4f04-8e70-44c375ccba55-kube-api-access-qt95m\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.220751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.220716    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:17.228656 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.228618    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.234591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.234556    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt95m\" (UniqueName: \"kubernetes.io/projected/658e5f59-662f-4f04-8e70-44c375ccba55-kube-api-access-qt95m\") pod \"maas-api-revoke-sub-c94c760d-4djb8\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.258131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.258093    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:17.608732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.608680    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"]
Nov 02 15:39:17.716606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:17.716564    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8" event={"ID":"658e5f59-662f-4f04-8e70-44c375ccba55","Type":"ContainerStarted","Data":"45c2c0772b87a272ef832dba2dec5fe8603c2332787e9620963e18e032ae2df5"}
Nov 02 15:39:18.725517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:18.725481    7019 generic.go:356] "Generic (PLEG): container finished" podID="658e5f59-662f-4f04-8e70-44c375ccba55" containerID="4a75748ba2c4e64443861cd0fb888a2e5e9f53652506813ef04880307bd3bdc5" exitCode=0
Nov 02 15:39:18.726010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:18.725543    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8" event={"ID":"658e5f59-662f-4f04-8e70-44c375ccba55","Type":"ContainerDied","Data":"4a75748ba2c4e64443861cd0fb888a2e5e9f53652506813ef04880307bd3bdc5"}
Nov 02 15:39:18.915881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:18.915847    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:19.923982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.923950    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:19.956151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.956120    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-openshift-ca\") pod \"658e5f59-662f-4f04-8e70-44c375ccba55\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") "
Nov 02 15:39:19.956350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.956193    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658e5f59-662f-4f04-8e70-44c375ccba55-kube-api-access-qt95m\" (UniqueName: \"kubernetes.io/projected/658e5f59-662f-4f04-8e70-44c375ccba55-kube-api-access-qt95m\") pod \"658e5f59-662f-4f04-8e70-44c375ccba55\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") "
Nov 02 15:39:19.956432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.956396    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-xks-ca\") pod \"658e5f59-662f-4f04-8e70-44c375ccba55\" (UID: \"658e5f59-662f-4f04-8e70-44c375ccba55\") "
Nov 02 15:39:19.956627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.956598    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-openshift-ca" pod "658e5f59-662f-4f04-8e70-44c375ccba55" (UID: "658e5f59-662f-4f04-8e70-44c375ccba55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:39:19.956896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.956877    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:19.958515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.958490    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-xks-ca" pod "658e5f59-662f-4f04-8e70-44c375ccba55" (UID: "658e5f59-662f-4f04-8e70-44c375ccba55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:19.958802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:19.958779    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658e5f59-662f-4f04-8e70-44c375ccba55-kube-api-access-qt95m" pod "658e5f59-662f-4f04-8e70-44c375ccba55" (UID: "658e5f59-662f-4f04-8e70-44c375ccba55"). InnerVolumeSpecName "kube-api-access-qt95m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:20.058574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:20.058495    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/658e5f59-662f-4f04-8e70-44c375ccba55-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:20.058574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:20.058523    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt95m\" (UniqueName: \"kubernetes.io/projected/658e5f59-662f-4f04-8e70-44c375ccba55-kube-api-access-qt95m\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:20.546712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:20.540253    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:20.747631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:20.747589    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"
Nov 02 15:39:20.747856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:20.747725    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8" event={"ID":"658e5f59-662f-4f04-8e70-44c375ccba55","Type":"ContainerDied","Data":"45c2c0772b87a272ef832dba2dec5fe8603c2332787e9620963e18e032ae2df5"}
Nov 02 15:39:20.747856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:20.747777    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c2c0772b87a272ef832dba2dec5fe8603c2332787e9620963e18e032ae2df5"
Nov 02 15:39:21.036450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:21.036412    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:39:21.036934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:21.036475    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:39:22.431808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:22.431766    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:22.707040 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:39:22.706944    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:39:22.767523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:22.767485    7019 generic.go:356] "Generic (PLEG): container finished" podID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerID="90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243" exitCode=0
Nov 02 15:39:22.767771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:22.767550    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" event={"ID":"9e68d4b4-3616-40af-86bc-d419e0d0fc0e","Type":"ContainerDied","Data":"90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243"}
Nov 02 15:39:22.768136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:22.768115    7019 scope.go:122] "RemoveContainer" containerID="90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243"
Nov 02 15:39:22.770000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:22.769968    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:22.770076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:22.770044    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:22.770122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:22.770075    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:23.800987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:23.800904    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" event={"ID":"9e68d4b4-3616-40af-86bc-d419e0d0fc0e","Type":"ContainerStarted","Data":"257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af"}
Nov 02 15:39:24.028838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:24.028171    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:24.140747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:24.140715    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:39:24.141097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:24.141069    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:39:24.141165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:24.141143    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 15:39:25.141454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:25.141410    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:39:25.141967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:25.141463    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 15:39:25.749971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:25.749937    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:26.141968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:26.141931    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:39:26.142374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:26.142001    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 15:39:27.141868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.141828    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:39:27.142027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.141898    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 15:39:27.352472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.352438    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"]
Nov 02 15:39:27.353119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.353102    7019 cpu_manager.go:414] "containerMap: removing container" podUID="658e5f59-662f-4f04-8e70-44c375ccba55" containerName="revoke-keys"
Nov 02 15:39:27.353184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.353120    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658e5f59-662f-4f04-8e70-44c375ccba55" containerName="revoke-keys"
Nov 02 15:39:27.353244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.353235    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="658e5f59-662f-4f04-8e70-44c375ccba55" containerName="revoke-keys"
Nov 02 15:39:27.356536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.356519    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.394506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.394411    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:39:27.394764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.394471    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:39:27.401788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.401575    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:27.410385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.410347    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"]
Nov 02 15:39:27.501768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.501720    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.502080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.502054    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t22rt\" (UniqueName: \"kubernetes.io/projected/affc85ff-e4d5-466a-983f-79e25be736d8-kube-api-access-t22rt\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.502221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.502151    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.603965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.603922    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22rt\" (UniqueName: \"kubernetes.io/projected/affc85ff-e4d5-466a-983f-79e25be736d8-kube-api-access-t22rt\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.604187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.604015    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.604187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.604117    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.604805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.604780    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.606733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.606709    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.628344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.628313    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22rt\" (UniqueName: \"kubernetes.io/projected/affc85ff-e4d5-466a-983f-79e25be736d8-kube-api-access-t22rt\") pod \"maas-api-revoke-sub-12780cf8-qr4qp\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:27.679259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:27.678878    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:28.020406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:28.020245    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"]
Nov 02 15:39:28.020781 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:39:28.020750    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffc85ff_e4d5_466a_983f_79e25be736d8.slice/crio-27563270b1e2ff6fa66fe2dcc464a08692fb5ed63cd70fefc9f832c3ba907782 WatchSource:0}: Error finding container 27563270b1e2ff6fa66fe2dcc464a08692fb5ed63cd70fefc9f832c3ba907782: Status 404 returned error can't find the container with id 27563270b1e2ff6fa66fe2dcc464a08692fb5ed63cd70fefc9f832c3ba907782
Nov 02 15:39:28.141779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:28.141740    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 15:39:28.141952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:28.141806    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 15:39:28.846085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:28.845986    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp" event={"ID":"affc85ff-e4d5-466a-983f-79e25be736d8","Type":"ContainerStarted","Data":"b6fd824c9ce202a296f0274d4940bbb70cebfdd967396b046b8aae0535c9af3f"}
Nov 02 15:39:28.846085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:28.846062    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp" event={"ID":"affc85ff-e4d5-466a-983f-79e25be736d8","Type":"ContainerStarted","Data":"27563270b1e2ff6fa66fe2dcc464a08692fb5ed63cd70fefc9f832c3ba907782"}
Nov 02 15:39:28.897289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:28.897230    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp" podStartSLOduration=1.897214768 podStartE2EDuration="1.897214768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:28.894732796 +0000 UTC m=+2563.148686420" watchObservedRunningTime="2026-11-02 15:39:28.897214768 +0000 UTC m=+2563.151168392"
Nov 02 15:39:28.952267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:28.952222    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:29.341161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:29.341123    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:39:29.342041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:29.342007    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:39:29.343637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:29.343613    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:39:29.854707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:29.854669    7019 generic.go:356] "Generic (PLEG): container finished" podID="affc85ff-e4d5-466a-983f-79e25be736d8" containerID="b6fd824c9ce202a296f0274d4940bbb70cebfdd967396b046b8aae0535c9af3f" exitCode=0
Nov 02 15:39:29.862413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:29.862383    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp" event={"ID":"affc85ff-e4d5-466a-983f-79e25be736d8","Type":"ContainerDied","Data":"b6fd824c9ce202a296f0274d4940bbb70cebfdd967396b046b8aae0535c9af3f"}
Nov 02 15:39:30.679916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:30.679852    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:31.032044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.032018    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:31.160444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.160415    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-openshift-ca\") pod \"affc85ff-e4d5-466a-983f-79e25be736d8\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") "
Nov 02 15:39:31.160687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.160557    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-xks-ca\") pod \"affc85ff-e4d5-466a-983f-79e25be736d8\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") "
Nov 02 15:39:31.160687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.160617    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affc85ff-e4d5-466a-983f-79e25be736d8-kube-api-access-t22rt\" (UniqueName: \"kubernetes.io/projected/affc85ff-e4d5-466a-983f-79e25be736d8-kube-api-access-t22rt\") pod \"affc85ff-e4d5-466a-983f-79e25be736d8\" (UID: \"affc85ff-e4d5-466a-983f-79e25be736d8\") "
Nov 02 15:39:31.160902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.160876    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-openshift-ca" pod "affc85ff-e4d5-466a-983f-79e25be736d8" (UID: "affc85ff-e4d5-466a-983f-79e25be736d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:39:31.161194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.161166    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:31.162750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.162728    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affc85ff-e4d5-466a-983f-79e25be736d8-kube-api-access-t22rt" pod "affc85ff-e4d5-466a-983f-79e25be736d8" (UID: "affc85ff-e4d5-466a-983f-79e25be736d8"). InnerVolumeSpecName "kube-api-access-t22rt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:31.162850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.162759    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-xks-ca" pod "affc85ff-e4d5-466a-983f-79e25be736d8" (UID: "affc85ff-e4d5-466a-983f-79e25be736d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:31.262512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.262433    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/affc85ff-e4d5-466a-983f-79e25be736d8-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:31.262512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.262465    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t22rt\" (UniqueName: \"kubernetes.io/projected/affc85ff-e4d5-466a-983f-79e25be736d8-kube-api-access-t22rt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:31.869824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.869788    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"
Nov 02 15:39:31.870035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.869821    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp" event={"ID":"affc85ff-e4d5-466a-983f-79e25be736d8","Type":"ContainerDied","Data":"27563270b1e2ff6fa66fe2dcc464a08692fb5ed63cd70fefc9f832c3ba907782"}
Nov 02 15:39:31.870035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:31.869871    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27563270b1e2ff6fa66fe2dcc464a08692fb5ed63cd70fefc9f832c3ba907782"
Nov 02 15:39:32.332318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:32.332232    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:33.912855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:33.912820    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:35.470030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:35.469962    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:37.089120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:37.089085    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:38.787324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:38.787284    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:39.522201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:39.522162    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:39.702095 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:39.702062    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:39.961635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:39.961596    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:40.357900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.357825    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"]
Nov 02 15:39:40.358739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.358713    7019 cpu_manager.go:414] "containerMap: removing container" podUID="affc85ff-e4d5-466a-983f-79e25be736d8" containerName="revoke-keys"
Nov 02 15:39:40.358739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.358736    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affc85ff-e4d5-466a-983f-79e25be736d8" containerName="revoke-keys"
Nov 02 15:39:40.358930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.358875    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="affc85ff-e4d5-466a-983f-79e25be736d8" containerName="revoke-keys"
Nov 02 15:39:40.363636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.363611    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.385919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.385893    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-28da58-openshift-default-dockercfg-lg4q9\""
Nov 02 15:39:40.402027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.401993    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"]
Nov 02 15:39:40.500869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.500819    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-envoy\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.500869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.500866    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1943de-bc74-4783-aeef-552f906a8f7a-istiod-ca-cert\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.501259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.500903    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-credential-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.501259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.500997    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-podinfo\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.501259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.501084    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.501259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.501199    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75fn\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-kube-api-access-j75fn\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.501491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.501301    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-certs\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.501491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.501343    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-data\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.501603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.501499    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-token\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.522532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.519892    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:40.607493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.607450    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-token\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.607809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.607786    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-envoy\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.607952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.607910    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1943de-bc74-4783-aeef-552f906a8f7a-istiod-ca-cert\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.608078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.608064    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-credential-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.608171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.608160    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-podinfo\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.608359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.608322    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.608510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.608494    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75fn\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-kube-api-access-j75fn\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.608677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.608639    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-certs\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.608834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.608813    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-data\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.609558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.609534    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-data\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.619914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.619886    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1943de-bc74-4783-aeef-552f906a8f7a-istiod-ca-cert\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.684770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.627993    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.684770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.628496    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-certs\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.684770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.628841    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-credential-socket\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.684770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.663683    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-podinfo\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.684770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.663697    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75fn\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-kube-api-access-j75fn\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.684770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.684320    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-token\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.701838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.701802    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-envoy\") pod \"e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") " pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:40.717232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.717196    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:40.765440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.765400    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"]
Nov 02 15:39:40.765701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.765675    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy" containerID="cri-o://c44adf0c3429bfe11c3732f0ad88a25903636785f285716aad02a5e219451f08" gracePeriod=30
Nov 02 15:39:40.988104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:40.988065    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:41.337863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:41.337833    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"]
Nov 02 15:39:41.338006 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:39:41.337944    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1943de_bc74_4783_aeef_552f906a8f7a.slice/crio-93fde9b33f4cde7e127534d9ac635a6783770dbf56ceee8fbf5e594a45e866b7 WatchSource:0}: Error finding container 93fde9b33f4cde7e127534d9ac635a6783770dbf56ceee8fbf5e594a45e866b7: Status 404 returned error can't find the container with id 93fde9b33f4cde7e127534d9ac635a6783770dbf56ceee8fbf5e594a45e866b7
Nov 02 15:39:41.341108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:41.341075    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:41.341190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:41.341142    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:41.341190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:41.341172    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:39:41.957878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:41.957840    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" event={"ID":"ad1943de-bc74-4783-aeef-552f906a8f7a","Type":"ContainerStarted","Data":"4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b"}
Nov 02 15:39:41.958078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:41.957904    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" event={"ID":"ad1943de-bc74-4783-aeef-552f906a8f7a","Type":"ContainerStarted","Data":"93fde9b33f4cde7e127534d9ac635a6783770dbf56ceee8fbf5e594a45e866b7"}
Nov 02 15:39:41.988998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:41.988958    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:42.010916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:42.010854    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podStartSLOduration=2.010836251 podStartE2EDuration="2.010836251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:42.008179399 +0000 UTC m=+2576.262133023" watchObservedRunningTime="2026-11-02 15:39:42.010836251 +0000 UTC m=+2576.264789876"
Nov 02 15:39:42.245290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:42.245196    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:42.998069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:42.998033    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:43.601683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:43.601609    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:43.977814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:43.977779    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:43.978828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:43.978802    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:39:44.034794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:44.033998    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:44.431768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:44.431731    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:44.962019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:44.961981    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:45.145397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:45.145353    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:45.709778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:45.709732    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:46.056215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.056125    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:39:46.087325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.087278    7019 generic.go:356] "Generic (PLEG): container finished" podID="67e54497-7979-492f-8d4c-6b3741b7d437" containerID="c44adf0c3429bfe11c3732f0ad88a25903636785f285716aad02a5e219451f08" exitCode=0
Nov 02 15:39:46.087495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.087351    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" event={"ID":"67e54497-7979-492f-8d4c-6b3741b7d437","Type":"ContainerDied","Data":"c44adf0c3429bfe11c3732f0ad88a25903636785f285716aad02a5e219451f08"}
Nov 02 15:39:46.087495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.087422    7019 scope.go:122] "RemoveContainer" containerID="adef7850b2a6e2f390c47d5cd24c9a2893bef01bbd1e6980717a999ac8c6cbcb"
Nov 02 15:39:46.228069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.228039    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 15:39:46.322182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322099    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67e54497-7979-492f-8d4c-6b3741b7d437-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e54497-7979-492f-8d4c-6b3741b7d437-istio-podinfo\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322254    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-istio-token\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-istio-token\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322306    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-socket\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322474    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-credential-socket\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322519    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-envoy\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322592    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-data\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322660    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-certs\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322694    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e54497-7979-492f-8d4c-6b3741b7d437-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e54497-7979-492f-8d4c-6b3741b7d437-istiod-ca-cert\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322761    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-kube-api-access-bwvbs\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-kube-api-access-bwvbs\") pod \"67e54497-7979-492f-8d4c-6b3741b7d437\" (UID: \"67e54497-7979-492f-8d4c-6b3741b7d437\") "
Nov 02 15:39:46.322947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.322914    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-credential-socket" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:46.323557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.323421    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.323557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.323467    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e54497-7979-492f-8d4c-6b3741b7d437-istiod-ca-cert" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:39:46.323557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.323529    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-certs" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:46.323924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.323708    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-data" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:46.324190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.324157    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-socket" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:46.325362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.325228    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67e54497-7979-492f-8d4c-6b3741b7d437-istio-podinfo" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:39:46.325848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.325822    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-istio-token" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:46.326364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.326333    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-kube-api-access-bwvbs" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "kube-api-access-bwvbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:46.326701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.326674    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-envoy" pod "67e54497-7979-492f-8d4c-6b3741b7d437" (UID: "67e54497-7979-492f-8d4c-6b3741b7d437"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.427942    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.427981    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.427996    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.428008    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67e54497-7979-492f-8d4c-6b3741b7d437-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.428022    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwvbs\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-kube-api-access-bwvbs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.428036    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67e54497-7979-492f-8d4c-6b3741b7d437-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.428048    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67e54497-7979-492f-8d4c-6b3741b7d437-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:46.429671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:46.428062    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67e54497-7979-492f-8d4c-6b3741b7d437-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:47.136084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.136043    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw" event={"ID":"67e54497-7979-492f-8d4c-6b3741b7d437","Type":"ContainerDied","Data":"a3ff9d1a456b0a74b32efbac89b7bdb16558f9c906f7eceeb8fe325e5d8c73d3"}
Nov 02 15:39:47.136371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.136352    7019 scope.go:122] "RemoveContainer" containerID="c44adf0c3429bfe11c3732f0ad88a25903636785f285716aad02a5e219451f08"
Nov 02 15:39:47.136581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.136566    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"
Nov 02 15:39:47.233718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.233679    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"]
Nov 02 15:39:47.246513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.246479    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-flxcw"]
Nov 02 15:39:47.482572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.482432    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:47.634674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.633788    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:47.856407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.856323    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:47.864349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:47.864315    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" path="/var/lib/kubelet/pods/67e54497-7979-492f-8d4c-6b3741b7d437/volumes"
Nov 02 15:39:48.773671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:48.773622    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:48.927154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:48.927122    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:49.565171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.565082    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:49.749108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.749051    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:49.996906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.996872    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"]
Nov 02 15:39:49.997684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.997664    7019 cpu_manager.go:414] "containerMap: removing container" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy"
Nov 02 15:39:49.997783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.997685    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy"
Nov 02 15:39:49.997783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.997744    7019 cpu_manager.go:414] "containerMap: removing container" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy"
Nov 02 15:39:49.997783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.997752    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy"
Nov 02 15:39:49.997958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:49.997930    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy"
Nov 02 15:39:50.004216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.004191    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.027228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.027204    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:39:50.038193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.038167    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"]
Nov 02 15:39:50.059184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.059153    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"]
Nov 02 15:39:50.060338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.060317    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e54497-7979-492f-8d4c-6b3741b7d437" containerName="istio-proxy"
Nov 02 15:39:50.064923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.064903    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.086109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.086081    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"]
Nov 02 15:39:50.099931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.099898    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6z8\" (UniqueName: \"kubernetes.io/projected/64b0db9e-7ac6-48ed-901a-6b817dbd3239-kube-api-access-cl6z8\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.100126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.100101    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.100206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.100185    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.127734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.127705    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:39:50.201523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.201484    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.201740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.201537    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.201740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.201670    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.201884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.201843    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzml\" (UniqueName: \"kubernetes.io/projected/01edccac-720b-4ee2-b39b-e54287931f64-kube-api-access-nvzml\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.201941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.201880    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.201941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.201933    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6z8\" (UniqueName: \"kubernetes.io/projected/64b0db9e-7ac6-48ed-901a-6b817dbd3239-kube-api-access-cl6z8\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.202484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.202456    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.204196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.204173    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.231215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.231186    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6z8\" (UniqueName: \"kubernetes.io/projected/64b0db9e-7ac6-48ed-901a-6b817dbd3239-kube-api-access-cl6z8\") pod \"maas-api-revoke-sub-ac81dddb-hhhsq\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.303554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.303456    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.303759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.303685    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzml\" (UniqueName: \"kubernetes.io/projected/01edccac-720b-4ee2-b39b-e54287931f64-kube-api-access-nvzml\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.303759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.303723    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.304361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.304324    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.306173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.306142    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.320388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.320362    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:50.327662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.327628    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzml\" (UniqueName: \"kubernetes.io/projected/01edccac-720b-4ee2-b39b-e54287931f64-kube-api-access-nvzml\") pod \"maas-api-revoke-sub-89747308-rg2lt\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.405893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.397000    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:50.635700 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:39:50.635668    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b0db9e_7ac6_48ed_901a_6b817dbd3239.slice/crio-4df189f4c1ecc9f317c97b785fbe2335f68e75bbc6315f156b10bdcc136f488c WatchSource:0}: Error finding container 4df189f4c1ecc9f317c97b785fbe2335f68e75bbc6315f156b10bdcc136f488c: Status 404 returned error can't find the container with id 4df189f4c1ecc9f317c97b785fbe2335f68e75bbc6315f156b10bdcc136f488c
Nov 02 15:39:50.636861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.636833    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"]
Nov 02 15:39:50.746664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.746620    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"]
Nov 02 15:39:50.748596 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:39:50.748565    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01edccac_720b_4ee2_b39b_e54287931f64.slice/crio-87a0d8ba37b638e00fcc6025d32689fd29b35532aced48f83d1d6f7eba224c93 WatchSource:0}: Error finding container 87a0d8ba37b638e00fcc6025d32689fd29b35532aced48f83d1d6f7eba224c93: Status 404 returned error can't find the container with id 87a0d8ba37b638e00fcc6025d32689fd29b35532aced48f83d1d6f7eba224c93
Nov 02 15:39:50.755811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.755284    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:50.912458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:50.912330    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:51.172784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:51.172752    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt" event={"ID":"01edccac-720b-4ee2-b39b-e54287931f64","Type":"ContainerStarted","Data":"87a0d8ba37b638e00fcc6025d32689fd29b35532aced48f83d1d6f7eba224c93"}
Nov 02 15:39:51.174882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:51.174857    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq" event={"ID":"64b0db9e-7ac6-48ed-901a-6b817dbd3239","Type":"ContainerStarted","Data":"4df189f4c1ecc9f317c97b785fbe2335f68e75bbc6315f156b10bdcc136f488c"}
Nov 02 15:39:51.396208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:51.396173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:52.200218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:52.200135    7019 generic.go:356] "Generic (PLEG): container finished" podID="01edccac-720b-4ee2-b39b-e54287931f64" containerID="95905fc27d55526d0688d8a5aeee9772e96e8eba6d878731b5fc3e5ee2b6a889" exitCode=0
Nov 02 15:39:52.200596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:52.200248    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt" event={"ID":"01edccac-720b-4ee2-b39b-e54287931f64","Type":"ContainerDied","Data":"95905fc27d55526d0688d8a5aeee9772e96e8eba6d878731b5fc3e5ee2b6a889"}
Nov 02 15:39:52.202524 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:52.202498    7019 generic.go:356] "Generic (PLEG): container finished" podID="64b0db9e-7ac6-48ed-901a-6b817dbd3239" containerID="48ced2708b8d49064c9503a6dd99ca723a0042cdcc9450dd5abab2349256ae5e" exitCode=0
Nov 02 15:39:52.202671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:52.202596    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq" event={"ID":"64b0db9e-7ac6-48ed-901a-6b817dbd3239","Type":"ContainerDied","Data":"48ced2708b8d49064c9503a6dd99ca723a0042cdcc9450dd5abab2349256ae5e"}
Nov 02 15:39:52.725050 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:39:52.724981    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:39:53.069841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.069745    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:53.420212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.420189    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:53.423204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.423181    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:53.453983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.453948    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64b0db9e-7ac6-48ed-901a-6b817dbd3239-kube-api-access-cl6z8\" (UniqueName: \"kubernetes.io/projected/64b0db9e-7ac6-48ed-901a-6b817dbd3239-kube-api-access-cl6z8\") pod \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") "
Nov 02 15:39:53.454114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454011    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01edccac-720b-4ee2-b39b-e54287931f64-kube-api-access-nvzml\" (UniqueName: \"kubernetes.io/projected/01edccac-720b-4ee2-b39b-e54287931f64-kube-api-access-nvzml\") pod \"01edccac-720b-4ee2-b39b-e54287931f64\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") "
Nov 02 15:39:53.454178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454156    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-openshift-ca\") pod \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") "
Nov 02 15:39:53.454226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454202    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-openshift-ca\") pod \"01edccac-720b-4ee2-b39b-e54287931f64\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") "
Nov 02 15:39:53.454280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454233    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-xks-ca\") pod \"01edccac-720b-4ee2-b39b-e54287931f64\" (UID: \"01edccac-720b-4ee2-b39b-e54287931f64\") "
Nov 02 15:39:53.454383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454367    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-xks-ca\") pod \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\" (UID: \"64b0db9e-7ac6-48ed-901a-6b817dbd3239\") "
Nov 02 15:39:53.454553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454530    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-openshift-ca" pod "64b0db9e-7ac6-48ed-901a-6b817dbd3239" (UID: "64b0db9e-7ac6-48ed-901a-6b817dbd3239"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:39:53.454625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454559    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-openshift-ca" pod "01edccac-720b-4ee2-b39b-e54287931f64" (UID: "01edccac-720b-4ee2-b39b-e54287931f64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:39:53.454991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454962    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:53.454991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.454982    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:53.456290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.456265    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-xks-ca" pod "01edccac-720b-4ee2-b39b-e54287931f64" (UID: "01edccac-720b-4ee2-b39b-e54287931f64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:53.456386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.456373    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-xks-ca" pod "64b0db9e-7ac6-48ed-901a-6b817dbd3239" (UID: "64b0db9e-7ac6-48ed-901a-6b817dbd3239"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:39:53.456743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.456719    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b0db9e-7ac6-48ed-901a-6b817dbd3239-kube-api-access-cl6z8" pod "64b0db9e-7ac6-48ed-901a-6b817dbd3239" (UID: "64b0db9e-7ac6-48ed-901a-6b817dbd3239"). InnerVolumeSpecName "kube-api-access-cl6z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:53.456820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.456754    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01edccac-720b-4ee2-b39b-e54287931f64-kube-api-access-nvzml" pod "01edccac-720b-4ee2-b39b-e54287931f64" (UID: "01edccac-720b-4ee2-b39b-e54287931f64"). InnerVolumeSpecName "kube-api-access-nvzml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:39:53.555749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.555715    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01edccac-720b-4ee2-b39b-e54287931f64-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:53.555749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.555744    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64b0db9e-7ac6-48ed-901a-6b817dbd3239-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:53.555749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.555753    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl6z8\" (UniqueName: \"kubernetes.io/projected/64b0db9e-7ac6-48ed-901a-6b817dbd3239-kube-api-access-cl6z8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:53.556068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:53.555762    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzml\" (UniqueName: \"kubernetes.io/projected/01edccac-720b-4ee2-b39b-e54287931f64-kube-api-access-nvzml\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:39:54.219779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.219693    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"
Nov 02 15:39:54.219939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.219692    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt" event={"ID":"01edccac-720b-4ee2-b39b-e54287931f64","Type":"ContainerDied","Data":"87a0d8ba37b638e00fcc6025d32689fd29b35532aced48f83d1d6f7eba224c93"}
Nov 02 15:39:54.219939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.219857    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a0d8ba37b638e00fcc6025d32689fd29b35532aced48f83d1d6f7eba224c93"
Nov 02 15:39:54.221335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.221317    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"
Nov 02 15:39:54.221335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.221326    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq" event={"ID":"64b0db9e-7ac6-48ed-901a-6b817dbd3239","Type":"ContainerDied","Data":"4df189f4c1ecc9f317c97b785fbe2335f68e75bbc6315f156b10bdcc136f488c"}
Nov 02 15:39:54.221485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.221362    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4df189f4c1ecc9f317c97b785fbe2335f68e75bbc6315f156b10bdcc136f488c"
Nov 02 15:39:54.486386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.486291    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:54.644169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.644117    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:54.866497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:54.866450    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:56.206514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:56.206480    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:56.409733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:56.409689    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:56.832690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:56.832613    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:39:58.504415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:39:58.504376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:00.118499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.118454    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:00.332043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.332004    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"]
Nov 02 15:40:00.332867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.332842    7019 cpu_manager.go:414] "containerMap: removing container" podUID="01edccac-720b-4ee2-b39b-e54287931f64" containerName="revoke-keys"
Nov 02 15:40:00.332990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.332871    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01edccac-720b-4ee2-b39b-e54287931f64" containerName="revoke-keys"
Nov 02 15:40:00.332990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.332893    7019 cpu_manager.go:414] "containerMap: removing container" podUID="64b0db9e-7ac6-48ed-901a-6b817dbd3239" containerName="revoke-keys"
Nov 02 15:40:00.332990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.332899    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64b0db9e-7ac6-48ed-901a-6b817dbd3239" containerName="revoke-keys"
Nov 02 15:40:00.333093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.333017    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="01edccac-720b-4ee2-b39b-e54287931f64" containerName="revoke-keys"
Nov 02 15:40:00.333093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.333028    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="64b0db9e-7ac6-48ed-901a-6b817dbd3239" containerName="revoke-keys"
Nov 02 15:40:00.349368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.349328    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.361121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.361084    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"]
Nov 02 15:40:00.367093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.367065    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:40:00.367413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.367386    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:40:00.475478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.475381    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"]
Nov 02 15:40:00.481282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.481246    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.500696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.500160    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.500696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.500285    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.500696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.500335    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lrf\" (UniqueName: \"kubernetes.io/projected/29a54855-6a17-40d4-bc1a-b01ac5377922-kube-api-access-k9lrf\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.509694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.505412    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"]
Nov 02 15:40:00.602153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.602110    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.602490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.602177    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lrf\" (UniqueName: \"kubernetes.io/projected/29a54855-6a17-40d4-bc1a-b01ac5377922-kube-api-access-k9lrf\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.602490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.602220    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d788da-4459-4c88-a032-abebe55690f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.602490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.602320    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d788da-4459-4c88-a032-abebe55690f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.602490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.602372    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qd6s\" (UniqueName: \"kubernetes.io/projected/43d788da-4459-4c88-a032-abebe55690f5-kube-api-access-7qd6s\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.603182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.602602    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.603598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.603573    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.606088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.606044    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.626508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.626475    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lrf\" (UniqueName: \"kubernetes.io/projected/29a54855-6a17-40d4-bc1a-b01ac5377922-kube-api-access-k9lrf\") pod \"maas-api-revoke-sub-2a966335-vz2zz\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.677365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.677329    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:00.704253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.704219    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d788da-4459-4c88-a032-abebe55690f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.704443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.704297    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d788da-4459-4c88-a032-abebe55690f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.704443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.704339    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qd6s\" (UniqueName: \"kubernetes.io/projected/43d788da-4459-4c88-a032-abebe55690f5-kube-api-access-7qd6s\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.705254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.705208    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d788da-4459-4c88-a032-abebe55690f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.707000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.706970    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d788da-4459-4c88-a032-abebe55690f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.729698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.729610    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qd6s\" (UniqueName: \"kubernetes.io/projected/43d788da-4459-4c88-a032-abebe55690f5-kube-api-access-7qd6s\") pod \"maas-api-revoke-sub-d1617c52-bvnvp\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.804731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.804233    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:00.995786 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:40:00.995751    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a54855_6a17_40d4_bc1a_b01ac5377922.slice/crio-bc10daeae4f8c72d543240d2ce6de7134e454818304aadf0248915f3425a369b WatchSource:0}: Error finding container bc10daeae4f8c72d543240d2ce6de7134e454818304aadf0248915f3425a369b: Status 404 returned error can't find the container with id bc10daeae4f8c72d543240d2ce6de7134e454818304aadf0248915f3425a369b
Nov 02 15:40:00.997060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:00.997030    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"]
Nov 02 15:40:01.116339 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:40:01.116308    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d788da_4459_4c88_a032_abebe55690f5.slice/crio-626fa4152a8a62c85b985d58c5fd907019391b960a13b289a808e944249bc85a WatchSource:0}: Error finding container 626fa4152a8a62c85b985d58c5fd907019391b960a13b289a808e944249bc85a: Status 404 returned error can't find the container with id 626fa4152a8a62c85b985d58c5fd907019391b960a13b289a808e944249bc85a
Nov 02 15:40:01.118604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:01.118579    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"]
Nov 02 15:40:01.278090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:01.278003    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp" event={"ID":"43d788da-4459-4c88-a032-abebe55690f5","Type":"ContainerStarted","Data":"626fa4152a8a62c85b985d58c5fd907019391b960a13b289a808e944249bc85a"}
Nov 02 15:40:01.279371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:01.279342    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz" event={"ID":"29a54855-6a17-40d4-bc1a-b01ac5377922","Type":"ContainerStarted","Data":"bc10daeae4f8c72d543240d2ce6de7134e454818304aadf0248915f3425a369b"}
Nov 02 15:40:01.819466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:01.815823    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:02.289261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:02.289223    7019 generic.go:356] "Generic (PLEG): container finished" podID="43d788da-4459-4c88-a032-abebe55690f5" containerID="664b1c66b3b3c3bd6e10f107ef280ead978f93b35c8e904e7229842b0fa59570" exitCode=0
Nov 02 15:40:02.289739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:02.289313    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp" event={"ID":"43d788da-4459-4c88-a032-abebe55690f5","Type":"ContainerDied","Data":"664b1c66b3b3c3bd6e10f107ef280ead978f93b35c8e904e7229842b0fa59570"}
Nov 02 15:40:02.291206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:02.291183    7019 generic.go:356] "Generic (PLEG): container finished" podID="29a54855-6a17-40d4-bc1a-b01ac5377922" containerID="5d36fd6cfe0f0bc929592e3b3883412ec5bf74599bec25731e34ce3c5f15798e" exitCode=0
Nov 02 15:40:02.291340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:02.291268    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz" event={"ID":"29a54855-6a17-40d4-bc1a-b01ac5377922","Type":"ContainerDied","Data":"5d36fd6cfe0f0bc929592e3b3883412ec5bf74599bec25731e34ce3c5f15798e"}
Nov 02 15:40:02.881942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:02.881907    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:03.361985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.361842    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:03.509059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.509030    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:03.540769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.540749    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:03.653747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.653717    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d788da-4459-4c88-a032-abebe55690f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d788da-4459-4c88-a032-abebe55690f5-maas-api-xks-ca\") pod \"43d788da-4459-4c88-a032-abebe55690f5\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") "
Nov 02 15:40:03.653921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.653781    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a54855-6a17-40d4-bc1a-b01ac5377922-kube-api-access-k9lrf\" (UniqueName: \"kubernetes.io/projected/29a54855-6a17-40d4-bc1a-b01ac5377922-kube-api-access-k9lrf\") pod \"29a54855-6a17-40d4-bc1a-b01ac5377922\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") "
Nov 02 15:40:03.653921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.653823    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-xks-ca\") pod \"29a54855-6a17-40d4-bc1a-b01ac5377922\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") "
Nov 02 15:40:03.653921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.653869    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-openshift-ca\") pod \"29a54855-6a17-40d4-bc1a-b01ac5377922\" (UID: \"29a54855-6a17-40d4-bc1a-b01ac5377922\") "
Nov 02 15:40:03.653921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.653910    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d788da-4459-4c88-a032-abebe55690f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d788da-4459-4c88-a032-abebe55690f5-maas-api-openshift-ca\") pod \"43d788da-4459-4c88-a032-abebe55690f5\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") "
Nov 02 15:40:03.654164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.654145    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d788da-4459-4c88-a032-abebe55690f5-kube-api-access-7qd6s\" (UniqueName: \"kubernetes.io/projected/43d788da-4459-4c88-a032-abebe55690f5-kube-api-access-7qd6s\") pod \"43d788da-4459-4c88-a032-abebe55690f5\" (UID: \"43d788da-4459-4c88-a032-abebe55690f5\") "
Nov 02 15:40:03.654329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.654292    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-openshift-ca" pod "29a54855-6a17-40d4-bc1a-b01ac5377922" (UID: "29a54855-6a17-40d4-bc1a-b01ac5377922"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:03.654440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.654368    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d788da-4459-4c88-a032-abebe55690f5-maas-api-openshift-ca" pod "43d788da-4459-4c88-a032-abebe55690f5" (UID: "43d788da-4459-4c88-a032-abebe55690f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:40:03.654805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.654787    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:40:03.654894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.654809    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d788da-4459-4c88-a032-abebe55690f5-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:40:03.656100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.656073    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-xks-ca" pod "29a54855-6a17-40d4-bc1a-b01ac5377922" (UID: "29a54855-6a17-40d4-bc1a-b01ac5377922"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:03.656275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.656260    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d788da-4459-4c88-a032-abebe55690f5-kube-api-access-7qd6s" pod "43d788da-4459-4c88-a032-abebe55690f5" (UID: "43d788da-4459-4c88-a032-abebe55690f5"). InnerVolumeSpecName "kube-api-access-7qd6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:03.656413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.656393    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a54855-6a17-40d4-bc1a-b01ac5377922-kube-api-access-k9lrf" pod "29a54855-6a17-40d4-bc1a-b01ac5377922" (UID: "29a54855-6a17-40d4-bc1a-b01ac5377922"). InnerVolumeSpecName "kube-api-access-k9lrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:40:03.656471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.656398    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d788da-4459-4c88-a032-abebe55690f5-maas-api-xks-ca" pod "43d788da-4459-4c88-a032-abebe55690f5" (UID: "43d788da-4459-4c88-a032-abebe55690f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:40:03.756150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.756117    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qd6s\" (UniqueName: \"kubernetes.io/projected/43d788da-4459-4c88-a032-abebe55690f5-kube-api-access-7qd6s\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:40:03.756150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.756147    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d788da-4459-4c88-a032-abebe55690f5-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:40:03.756150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.756157    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lrf\" (UniqueName: \"kubernetes.io/projected/29a54855-6a17-40d4-bc1a-b01ac5377922-kube-api-access-k9lrf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:40:03.756394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:03.756169    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a54855-6a17-40d4-bc1a-b01ac5377922-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:40:04.310946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:04.310914    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"
Nov 02 15:40:04.311126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:04.310910    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp" event={"ID":"43d788da-4459-4c88-a032-abebe55690f5","Type":"ContainerDied","Data":"626fa4152a8a62c85b985d58c5fd907019391b960a13b289a808e944249bc85a"}
Nov 02 15:40:04.311126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:04.311059    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="626fa4152a8a62c85b985d58c5fd907019391b960a13b289a808e944249bc85a"
Nov 02 15:40:04.312442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:04.312423    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"
Nov 02 15:40:04.312442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:04.312438    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz" event={"ID":"29a54855-6a17-40d4-bc1a-b01ac5377922","Type":"ContainerDied","Data":"bc10daeae4f8c72d543240d2ce6de7134e454818304aadf0248915f3425a369b"}
Nov 02 15:40:04.312638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:04.312487    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc10daeae4f8c72d543240d2ce6de7134e454818304aadf0248915f3425a369b"
Nov 02 15:40:04.958396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:04.958361    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:06.044094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:06.044060    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"]
Nov 02 15:40:06.058451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:06.058404    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fcf0be2-9qqj2"]
Nov 02 15:40:06.605158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:06.605120    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:07.838167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:07.838123    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:07.863029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:07.862995    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0" path="/var/lib/kubelet/pods/8dc2fac3-fc3c-4cc7-ac50-42a48009ceb0/volumes"
Nov 02 15:40:08.338059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:08.338025    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:09.956930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:09.956900    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:11.639376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:11.639343    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:13.274629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:13.274590    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:14.864855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:14.864821    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:15.043474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:15.042093    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:15.279185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:15.279077    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:16.312502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:16.312457    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:16.966259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:16.966221    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:18.035851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:18.035817    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"]
Nov 02 15:40:18.048182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:18.048155    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96d9d05a-w6fw7"]
Nov 02 15:40:18.463156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:18.463123    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:19.863060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:19.863029    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06357e76-a190-4dba-9cbc-4a6b32662f89" path="/var/lib/kubelet/pods/06357e76-a190-4dba-9cbc-4a6b32662f89/volumes"
Nov 02 15:40:20.049308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:20.049271    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:21.711565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:21.711509    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:22.708500 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:40:22.708455    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:40:23.319207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:23.319172    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:23.476459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:23.476424    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:24.890373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:24.890338    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:26.622598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:26.622171    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:28.435944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:28.435885    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:29.479711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:29.479671    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:30.078341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:30.078247    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:31.756306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:31.756261    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:31.911900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:31.911856    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:32.081396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:32.081268    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:33.496523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:33.496488    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:36.553212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:36.553178    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:37.041258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:37.041211    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:37.372727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:37.372692    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:37.959296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:37.959266    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:38.507636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:38.507381    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:38.684238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:38.684204    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:39.012069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:39.012032    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:40.603186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:40.603146    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:42.126928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:42.126885    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:42.315540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:42.315501    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:43.660819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:43.660779    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:40:43.661342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:43.660858    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:40:43.925173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:43.925071    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:44.097098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:44.097060    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:44.301731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:44.274089    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:45.764070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:45.764037    7019 generic.go:356] "Generic (PLEG): container finished" podID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerID="d7fd9341859c315bf8a6ff50b0c28a7c84cb373ef80398ef7df0f0a1abc8de41" exitCode=0
Nov 02 15:40:45.764517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:45.764112    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" event={"ID":"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf","Type":"ContainerDied","Data":"d7fd9341859c315bf8a6ff50b0c28a7c84cb373ef80398ef7df0f0a1abc8de41"}
Nov 02 15:40:45.764591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:45.764576    7019 scope.go:122] "RemoveContainer" containerID="d7fd9341859c315bf8a6ff50b0c28a7c84cb373ef80398ef7df0f0a1abc8de41"
Nov 02 15:40:45.766350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:45.766317    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:40:45.766479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:45.766395    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:40:45.766479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:45.766449    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:40:45.960091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:45.960041    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:46.234516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.230730    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"]
Nov 02 15:40:46.234516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.231630    7019 cpu_manager.go:414] "containerMap: removing container" podUID="29a54855-6a17-40d4-bc1a-b01ac5377922" containerName="revoke-keys"
Nov 02 15:40:46.234516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.231668    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a54855-6a17-40d4-bc1a-b01ac5377922" containerName="revoke-keys"
Nov 02 15:40:46.234516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.231726    7019 cpu_manager.go:414] "containerMap: removing container" podUID="43d788da-4459-4c88-a032-abebe55690f5" containerName="revoke-keys"
Nov 02 15:40:46.234516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.231734    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d788da-4459-4c88-a032-abebe55690f5" containerName="revoke-keys"
Nov 02 15:40:46.234516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.231870    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d788da-4459-4c88-a032-abebe55690f5" containerName="revoke-keys"
Nov 02 15:40:46.234516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.231887    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a54855-6a17-40d4-bc1a-b01ac5377922" containerName="revoke-keys"
Nov 02 15:40:46.235869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.235844    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.255695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.254933    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"default-dockercfg-sn8s6\""
Nov 02 15:40:46.255695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.255392    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"]
Nov 02 15:40:46.264130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.264103    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"sub-test-model-fda646-kserve-self-signed-certs\""
Nov 02 15:40:46.264299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.264261    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"kube-root-ca.crt\""
Nov 02 15:40:46.264384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.264314    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-fda646\"/\"openshift-service-ca.crt\""
Nov 02 15:40:46.273167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.273134    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tls-certs\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.273347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.273321    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-home\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.273437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.273385    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-dshm\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.273437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.273416    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrnc\" (UniqueName: \"kubernetes.io/projected/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-kube-api-access-vvrnc\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.273571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.273509    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-model-cache\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.273673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.273632    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tmp-dir\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.319508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.319472    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:46.375462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.375419    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-dshm\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.375700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.375472    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrnc\" (UniqueName: \"kubernetes.io/projected/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-kube-api-access-vvrnc\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.375700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.375531    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-model-cache\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.375700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.375674    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tmp-dir\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.375907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.375778    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tls-certs\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.375907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.375894    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-home\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.376248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.376219    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-model-cache\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.376348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.376297    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-home\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.376612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.376590    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tmp-dir\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.378010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.377990    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-dshm\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.379017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.378994    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tls-certs\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.404744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.404718    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrnc\" (UniqueName: \"kubernetes.io/projected/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-kube-api-access-vvrnc\") pod \"sub-test-model-fda646-kserve-78d5bf6579-vs676\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") " pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.554226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.553909    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:46.776836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.776800    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" event={"ID":"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf","Type":"ContainerStarted","Data":"7e590cf1ca2ae9df7d72225431ea5b2e785d68c3a65c222a2180f06f8a263d13"}
Nov 02 15:40:46.911858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:46.911821    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"]
Nov 02 15:40:46.919449 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:40:46.919415    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03fd0c4c_f425_4245_a6a2_3b9af4cd3fb4.slice/crio-734c026fd64498e476f31df357c14cc3734b6b1169a84fe53358f50523224953 WatchSource:0}: Error finding container 734c026fd64498e476f31df357c14cc3734b6b1169a84fe53358f50523224953: Status 404 returned error can't find the container with id 734c026fd64498e476f31df357c14cc3734b6b1169a84fe53358f50523224953
Nov 02 15:40:47.035697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.035667    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"]
Nov 02 15:40:47.041835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.041809    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45a38853-pw95j"]
Nov 02 15:40:47.562694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.562644    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:40:47.562925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.562902    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 02 15:40:47.562998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.562951    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 02 15:40:47.598904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.598867    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:47.803669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.803560    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" event={"ID":"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4","Type":"ContainerStarted","Data":"047da94fe54fd066472ac19cecfb794614adb064c24c6326e6ed7a9906422db9"}
Nov 02 15:40:47.803669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.803609    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" event={"ID":"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4","Type":"ContainerStarted","Data":"734c026fd64498e476f31df357c14cc3734b6b1169a84fe53358f50523224953"}
Nov 02 15:40:47.848553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.848429    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" podStartSLOduration=1.848411965 podStartE2EDuration="1.848411965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:47.847514535 +0000 UTC m=+2642.101468159" watchObservedRunningTime="2026-11-02 15:40:47.848411965 +0000 UTC m=+2642.102365590"
Nov 02 15:40:47.862882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:47.862851    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3" path="/var/lib/kubelet/pods/0416c2b7-415d-4c4e-9f9f-bfe84ef19fc3/volumes"
Nov 02 15:40:48.262267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.262234    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"]
Nov 02 15:40:48.278908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.278878    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.311072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.311039    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"]
Nov 02 15:40:48.322354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.315120    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6f2e0c\"/\"sub-test-model-6f2e0c-kserve-self-signed-certs\""
Nov 02 15:40:48.328907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.328880    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6f2e0c\"/\"openshift-service-ca.crt\""
Nov 02 15:40:48.329063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.328995    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6f2e0c\"/\"default-dockercfg-cjfhd\""
Nov 02 15:40:48.329145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.329094    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6f2e0c\"/\"kube-root-ca.crt\""
Nov 02 15:40:48.359675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.359483    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:48.425770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.425735    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-home\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.425770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.425771    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-model-cache\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.426014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.425799    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tls-certs\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.426014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.425855    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-dshm\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.426014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.425873    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tmp-dir\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.426014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.425945    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j9cl\" (UniqueName: \"kubernetes.io/projected/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-kube-api-access-4j9cl\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.528523    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-home\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.529061    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-model-cache\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.529117    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tls-certs\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.529213    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-dshm\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.529244    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tmp-dir\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.529373    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j9cl\" (UniqueName: \"kubernetes.io/projected/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-kube-api-access-4j9cl\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.530394    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-model-cache\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.529001    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-home\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.532959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.532149    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tmp-dir\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.539668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.536278    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tls-certs\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.556670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.556370    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:48.563671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.562222    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 02 15:40:48.563671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.562280    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 02 15:40:48.573669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.566635    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-dshm\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.581670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.581606    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j9cl\" (UniqueName: \"kubernetes.io/projected/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-kube-api-access-4j9cl\") pod \"sub-test-model-6f2e0c-kserve-75d6b86489-92qkm\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") " pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:48.639674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:48.631921    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:49.016935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:49.016857    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"]
Nov 02 15:40:49.023084 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:40:49.023055    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32f7da44_9c7b_42fc_8b24_89f61b2df9e7.slice/crio-29e9c027efd13d2212c5c82afded158ddc6a60f6424d5094a393c96082c9d5c4 WatchSource:0}: Error finding container 29e9c027efd13d2212c5c82afded158ddc6a60f6424d5094a393c96082c9d5c4: Status 404 returned error can't find the container with id 29e9c027efd13d2212c5c82afded158ddc6a60f6424d5094a393c96082c9d5c4
Nov 02 15:40:49.180563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:49.180526    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:49.536281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:49.536218    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:49.562693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:49.562627    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 02 15:40:49.562835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:49.562709    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 02 15:40:49.835585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:49.835497    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm" event={"ID":"32f7da44-9c7b-42fc-8b24-89f61b2df9e7","Type":"ContainerStarted","Data":"bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789"}
Nov 02 15:40:49.835585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:49.835548    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm" event={"ID":"32f7da44-9c7b-42fc-8b24-89f61b2df9e7","Type":"ContainerStarted","Data":"29e9c027efd13d2212c5c82afded158ddc6a60f6424d5094a393c96082c9d5c4"}
Nov 02 15:40:50.562114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:50.562082    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 02 15:40:50.562504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:50.562144    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 02 15:40:50.639200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:50.639169    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:50.913133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:50.913093    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:51.562000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:51.561967    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body=
Nov 02 15:40:51.562204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:51.562031    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused"
Nov 02 15:40:52.031772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.031709    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm" podStartSLOduration=4.031692983 podStartE2EDuration="4.031692983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:49.893886641 +0000 UTC m=+2644.147840267" watchObservedRunningTime="2026-11-02 15:40:52.031692983 +0000 UTC m=+2646.285646641"
Nov 02 15:40:52.032520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.032496    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"]
Nov 02 15:40:52.045131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.045104    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e564c91-vbt84"]
Nov 02 15:40:52.489275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.489244    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:52.567192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.567161    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:40:52.567616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.567545    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:40:52.568216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.568195    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:40:52.717310 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:40:52.717268    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:40:52.974925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:52.974888    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:53.143409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:53.143375    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:53.863366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:53.863327    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf9945f-a963-4e3b-b40b-bdcc5d117f0e" path="/var/lib/kubelet/pods/bcf9945f-a963-4e3b-b40b-bdcc5d117f0e/volumes"
Nov 02 15:40:54.104197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:54.104160    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:55.643311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:55.643266    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:56.555504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:56.555468    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:56.569020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:56.568981    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:56.906124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:56.906093    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:40:57.293858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:57.293774    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:58.639474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:58.639438    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:58.652408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:58.652382    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:58.923797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:58.923715    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:40:59.183882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.183788    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:40:59.513934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.513844    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"]
Nov 02 15:40:59.519949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.519924    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.537234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.537213    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-b55da1-openshift-default-dockercfg-rpg55\""
Nov 02 15:40:59.553504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.553472    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"]
Nov 02 15:40:59.610466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.610431    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-data\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.610675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.610513    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8gm8\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-kube-api-access-m8gm8\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.610675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.610572    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.610783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.610643    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-credential-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.610840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.610800    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-certs\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.610916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.610891    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-envoy\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.610977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.610945    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-token\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.611039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.611021    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-podinfo\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.611101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.611087    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.712708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.712634    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8gm8\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-kube-api-access-m8gm8\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.712727    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.712754    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-credential-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.712839    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-certs\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.712890    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-envoy\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.712925    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-token\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.713044    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-podinfo\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.713151    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.713202    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-data\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.713626    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istiod-ca-cert\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.713701    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-certs\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.713774    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.713836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.713800    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-credential-socket\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.714055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.714032    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-data\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.715380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.715355    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-envoy\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.715708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.715684    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-podinfo\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.739339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.739296    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8gm8\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-kube-api-access-m8gm8\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.740280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.740257    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-token\") pod \"e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") " pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:40:59.834864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:40:59.834774    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:41:00.173128 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:41:00.173095    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef64b9b_b435_4fde_9efa_a3e7429dd1f2.slice/crio-e717cf4ed0f5e0299c2543088a49daa47d8a818538ff3ae1af6f441f25d23775 WatchSource:0}: Error finding container e717cf4ed0f5e0299c2543088a49daa47d8a818538ff3ae1af6f441f25d23775: Status 404 returned error can't find the container with id e717cf4ed0f5e0299c2543088a49daa47d8a818538ff3ae1af6f441f25d23775
Nov 02 15:41:00.174758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:00.174671    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"]
Nov 02 15:41:00.176332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:00.176289    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:00.176425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:00.176377    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:00.176466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:00.176421    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:00.801950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:00.801902    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:00.946692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:00.946633    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" event={"ID":"cef64b9b-b435-4fde-9efa-a3e7429dd1f2","Type":"ContainerStarted","Data":"702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab"}
Nov 02 15:41:00.946869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:00.946717    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" event={"ID":"cef64b9b-b435-4fde-9efa-a3e7429dd1f2","Type":"ContainerStarted","Data":"e717cf4ed0f5e0299c2543088a49daa47d8a818538ff3ae1af6f441f25d23775"}
Nov 02 15:41:01.835279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:01.835244    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:41:01.836838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:01.836807    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 15:41:01.836980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:01.836861    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 15:41:02.492754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:02.489975    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:02.836091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:02.836014    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 15:41:02.836091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:02.836066    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 15:41:03.020601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:03.020560    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:03.835816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:03.835776    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 15:41:03.836004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:03.835842    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 15:41:04.221035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:04.220996    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:04.835801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:04.835765    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 15:41:04.836021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:04.835828    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 15:41:05.536021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:05.535979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:05.697723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:05.697691    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:05.836052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:05.835960    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 15:41:05.836052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:05.836030    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 15:41:05.838569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:05.838542    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:06.840368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:06.840336    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:41:06.840822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:06.840744    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:41:06.841527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:06.841508    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:41:06.902666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:06.902607    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" podStartSLOduration=7.9025934509999995 podStartE2EDuration="7.902593451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:01.001007188 +0000 UTC m=+2655.254960813" watchObservedRunningTime="2026-11-02 15:41:06.902593451 +0000 UTC m=+2661.156547079"
Nov 02 15:41:07.502955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:07.502916    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:08.025438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:08.025398    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:09.315575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:09.315536    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:11.005642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:11.005605    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:12.590535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:12.590484    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:14.272986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:14.272943    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:15.043084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:15.043049    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"]
Nov 02 15:41:15.057239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:15.057209    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f407b067-hcw4t"]
Nov 02 15:41:15.762603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:15.762569    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:15.864483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:15.864448    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500fa23d-ea61-441a-9e7e-62dacd83331b" path="/var/lib/kubelet/pods/500fa23d-ea61-441a-9e7e-62dacd83331b/volumes"
Nov 02 15:41:15.937531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:15.937499    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:16.811263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:16.811219    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:17.786807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:17.786758    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:18.050386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:18.050307    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"]
Nov 02 15:41:18.061499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:18.061471    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69bdd0d-6xr6f"]
Nov 02 15:41:19.455475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:19.455437    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:19.862795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:19.862764    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f031aa4e-da90-4dae-a7b4-e7124fb67d84" path="/var/lib/kubelet/pods/f031aa4e-da90-4dae-a7b4-e7124fb67d84/volumes"
Nov 02 15:41:21.011703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:21.011634    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:21.263317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:21.263227    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:22.707847 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:41:22.707803    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:41:22.813636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:22.813600    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:23.646123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:23.646074    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:41:24.462392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:24.462353    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:25.064325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:25.064285    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"]
Nov 02 15:41:25.077789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:25.077563    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e83dc05e-fcl76"]
Nov 02 15:41:25.863782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:25.863746    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c989a692-390f-4ecf-a85b-bb0acf95adf9" path="/var/lib/kubelet/pods/c989a692-390f-4ecf-a85b-bb0acf95adf9/volumes"
Nov 02 15:41:26.216081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:26.215983    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:27.685120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:27.685088    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:28.091423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:28.091332    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:29.809181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:29.809145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:31.497598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:31.497475    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:33.136555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:33.136491    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:35.010223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:35.010187    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:35.298325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:35.298242    7019 generic.go:356] "Generic (PLEG): container finished" podID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerID="4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b" exitCode=0
Nov 02 15:41:35.298485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:35.298313    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" event={"ID":"ad1943de-bc74-4783-aeef-552f906a8f7a","Type":"ContainerDied","Data":"4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b"}
Nov 02 15:41:35.298794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:35.298779    7019 scope.go:122] "RemoveContainer" containerID="4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b"
Nov 02 15:41:35.300566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:35.300537    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:35.300636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:35.300610    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:35.300690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:35.300661    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:36.310017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:36.309978    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" event={"ID":"ad1943de-bc74-4783-aeef-552f906a8f7a","Type":"ContainerStarted","Data":"9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477"}
Nov 02 15:41:36.582178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:36.582092    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:36.990021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:36.989983    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:41:36.990775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:36.990295    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:41:36.990973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:36.990952    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 02 15:41:38.001091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.001046    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:41:38.001617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.001114    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 02 15:41:38.295098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.295002    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:38.742076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.742046    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"]
Nov 02 15:41:38.745712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.745686    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.784676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.782592    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"]
Nov 02 15:41:38.784676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.782827    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-6f2e0c-openshift-default-dockercfg-mpw2f\""
Nov 02 15:41:38.925581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925545    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.925581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925590    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.925816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925609    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.925816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925686    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7d101b-466f-453a-a822-29dc40406ebb-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.925816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925744    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7d101b-466f-453a-a822-29dc40406ebb-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.925927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925873    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.925927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925891    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.926021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.925938    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.926056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.926035    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6z8\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-kube-api-access-jw6z8\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:38.991672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.991119    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:41:38.991672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:38.991202    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 02 15:41:39.027706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.027614    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6z8\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-kube-api-access-jw6z8\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.027779    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.027825    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.027853    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.027877    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7d101b-466f-453a-a822-29dc40406ebb-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.027903    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7d101b-466f-453a-a822-29dc40406ebb-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028090    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028142    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028228    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028264    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-data\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028314    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028406    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-credential-socket\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028606    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-certs\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.028939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.028921    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7d101b-466f-453a-a822-29dc40406ebb-istiod-ca-cert\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.030501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.030481    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-envoy\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.030631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.030523    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7d101b-466f-453a-a822-29dc40406ebb-istio-podinfo\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.055024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.054997    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-istio-token\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.055330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.055309    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6z8\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-kube-api-access-jw6z8\") pod \"e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") " pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.059521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.059497    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:39.429910 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:41:39.429871    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7d101b_466f_453a_a822_29dc40406ebb.slice/crio-17ba3a82bd93ee37c7d11ad3a37621f7f921f3386d39df2931d9d7a0c5f07113 WatchSource:0}: Error finding container 17ba3a82bd93ee37c7d11ad3a37621f7f921f3386d39df2931d9d7a0c5f07113: Status 404 returned error can't find the container with id 17ba3a82bd93ee37c7d11ad3a37621f7f921f3386d39df2931d9d7a0c5f07113
Nov 02 15:41:39.430405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.430380    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"]
Nov 02 15:41:39.432754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.432720    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:39.432848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.432783    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:39.432848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.432810    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:41:39.989440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.989000    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:41:39.989440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:39.989062    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 02 15:41:40.012028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:40.011983    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:40.361184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:40.361152    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" event={"ID":"9c7d101b-466f-453a-a822-29dc40406ebb","Type":"ContainerStarted","Data":"8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b"}
Nov 02 15:41:40.361581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:40.361210    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" event={"ID":"9c7d101b-466f-453a-a822-29dc40406ebb","Type":"ContainerStarted","Data":"17ba3a82bd93ee37c7d11ad3a37621f7f921f3386d39df2931d9d7a0c5f07113"}
Nov 02 15:41:40.428202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:40.428146    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podStartSLOduration=2.428132169 podStartE2EDuration="2.428132169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:40.427019288 +0000 UTC m=+2694.680972916" watchObservedRunningTime="2026-11-02 15:41:40.428132169 +0000 UTC m=+2694.682085792"
Nov 02 15:41:40.991198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:40.991151    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 02 15:41:40.991579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:40.991542    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 02 15:41:41.060609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:41.060571    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:41.062201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:41.062176    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 15:41:41.062339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:41.062220    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 15:41:41.648677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:41.647435    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:42.001727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:42.001519    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:41:42.002727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:42.002699    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:41:42.002968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:42.002949    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:41:42.060085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:42.060051    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 15:41:42.060277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:42.060108    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 15:41:43.060198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:43.060163    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 15:41:43.060608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:43.060223    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 15:41:43.275701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:43.272534    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:44.060992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:44.060953    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 15:41:44.061519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:44.061021    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 15:41:44.430192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:44.430144    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:44.629217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:44.629075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:45.060853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:45.060818    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 15:41:45.061027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:45.060868    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 15:41:45.473695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:45.473638    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:46.060928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:46.060892    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 02 15:41:46.061111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:46.060970    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 02 15:41:47.064592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:47.064563    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:47.065018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:47.064987    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:47.065768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:47.065747    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:41:47.136427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:47.136386    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:48.814677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:48.813507    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:49.322444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:49.322411    7019 scope.go:122] "RemoveContainer" containerID="b807e1b17a320ea0ac2628e27ea9ba132553ebd98c53bc1e89f11f3bd9e9bfa5"
Nov 02 15:41:49.333067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:49.333047    7019 scope.go:122] "RemoveContainer" containerID="fca7c47491d222d846931efcbb7579d08751e53c00c9bbdcd40a10ec501015fd"
Nov 02 15:41:49.344410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:49.344389    7019 scope.go:122] "RemoveContainer" containerID="810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40"
Nov 02 15:41:49.354317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:49.354295    7019 scope.go:122] "RemoveContainer" containerID="6ac344f23baefb76ab41efaca84784cbaf0fbc2d3a6332a7e52cc5c3d281c631"
Nov 02 15:41:49.364114 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:41:49.364087    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40\": container with ID starting with 810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40 not found: ID does not exist" containerID="810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40"
Nov 02 15:41:49.364219 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:41:49.364129    7019 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40\": container with ID starting with 810c30125142b17d26cd7100fa2d2ac58a086c8b41c234e10786464aef721b40 not found: ID does not exist"
Nov 02 15:41:49.367894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:49.367865    7019 scope.go:122] "RemoveContainer" containerID="b8647cffd623a0cf45312fae714ee8aea8ec440ecea16d7aa25f1fbd1a5e0cea"
Nov 02 15:41:49.670127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:49.670093    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:50.034722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:50.034621    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"]
Nov 02 15:41:50.047455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:50.047428    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3e6873-vsl79"]
Nov 02 15:41:50.367608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:50.367569    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:51.865439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:51.865407    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7dc0f1-0a16-41ab-99cb-478903dddeb2" path="/var/lib/kubelet/pods/7a7dc0f1-0a16-41ab-99cb-478903dddeb2/volumes"
Nov 02 15:41:52.004674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.004603    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"]
Nov 02 15:41:52.010737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.010706    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.027136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.018431    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:52.033409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.033338    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f6f1a0\"/\"rate-test-model-f6f1a0-kserve-self-signed-certs\""
Nov 02 15:41:52.046782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.046701    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f6f1a0\"/\"default-dockercfg-6qtqr\""
Nov 02 15:41:52.046782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.046737    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f6f1a0\"/\"openshift-service-ca.crt\""
Nov 02 15:41:52.047039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.046802    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f6f1a0\"/\"kube-root-ca.crt\""
Nov 02 15:41:52.050222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.050194    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"]
Nov 02 15:41:52.113301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.113256    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-home\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.113519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.113341    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-dshm\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.113519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.113447    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-tmp-dir\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.113519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.113513    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-model-cache\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.113768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.113740    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b712526-9385-4c26-b37e-8b0bee370de1-tls-certs\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.113839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.113824    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdv6\" (UniqueName: \"kubernetes.io/projected/3b712526-9385-4c26-b37e-8b0bee370de1-kube-api-access-rwdv6\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.183267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.183171    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:52.215591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.215548    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b712526-9385-4c26-b37e-8b0bee370de1-tls-certs\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.215806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.215618    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdv6\" (UniqueName: \"kubernetes.io/projected/3b712526-9385-4c26-b37e-8b0bee370de1-kube-api-access-rwdv6\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.215806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.215767    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-home\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.215955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.215830    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-dshm\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.215955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.215880    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-tmp-dir\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.215955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.215951    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-model-cache\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.216255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.216227    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-home\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.216356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.216309    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-tmp-dir\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.216448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.216422    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-model-cache\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.218513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.218491    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b712526-9385-4c26-b37e-8b0bee370de1-tls-certs\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.218961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.218936    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-dshm\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.240317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.240285    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdv6\" (UniqueName: \"kubernetes.io/projected/3b712526-9385-4c26-b37e-8b0bee370de1-kube-api-access-rwdv6\") pod \"rate-test-model-f6f1a0-kserve-855549489c-8l2w2\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") " pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.352970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.352926    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:52.706094 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:41:52.706061    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b712526_9385_4c26_b37e_8b0bee370de1.slice/crio-27b5487d241afcd389b3f9b25c084e7ec65cbf3dc6f313ad4c304dae3100b313 WatchSource:0}: Error finding container 27b5487d241afcd389b3f9b25c084e7ec65cbf3dc6f313ad4c304dae3100b313: Status 404 returned error can't find the container with id 27b5487d241afcd389b3f9b25c084e7ec65cbf3dc6f313ad4c304dae3100b313
Nov 02 15:41:52.708275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:52.708246    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"]
Nov 02 15:41:52.715836 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:41:52.715802    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:41:53.520774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:53.520718    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2" event={"ID":"3b712526-9385-4c26-b37e-8b0bee370de1","Type":"ContainerStarted","Data":"97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052"}
Nov 02 15:41:53.521295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:53.520801    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2" event={"ID":"3b712526-9385-4c26-b37e-8b0bee370de1","Type":"ContainerStarted","Data":"27b5487d241afcd389b3f9b25c084e7ec65cbf3dc6f313ad4c304dae3100b313"}
Nov 02 15:41:53.567905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:53.567852    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2" podStartSLOduration=2.567839425 podStartE2EDuration="2.567839425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:53.565017137 +0000 UTC m=+2707.818970794" watchObservedRunningTime="2026-11-02 15:41:53.567839425 +0000 UTC m=+2707.821793048"
Nov 02 15:41:53.705163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:53.705123    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:54.354468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:54.354436    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:41:55.285570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:55.285534    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:57.018717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:57.013394    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:57.181713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:57.180717    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:41:58.724864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:41:58.724799    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:00.706029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:00.700165    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:02.354334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:02.354297    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:42:02.368569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:02.368529    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:42:02.442181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:02.442110    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:02.610993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:02.610968    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:42:03.099463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:03.099385    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:03.336442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:03.336408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:03.627671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:03.624478    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:03.823814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:03.823462    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:04.227204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:04.227166    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:06.055350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:06.055310    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:07.788261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:07.788224    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:08.055720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:08.055562    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:08.265319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:08.265274    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:09.389171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:09.389133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:09.662576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:09.662475    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:09.820467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:09.820408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:10.416150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:10.416110    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:10.623466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:10.623420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:10.821219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:10.821130    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:10.977124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:10.977064    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:11.274147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:11.274111    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:11.835126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:11.835089    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:13.480257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:13.480217    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:14.942864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:14.942823    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:15.156892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:15.156857    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:15.429432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:15.429392    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:16.694197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:16.694160    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:16.919192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:16.919153    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:17.497264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:17.497229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:19.209699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:19.205601    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:20.711110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:20.711074    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:22.411559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:22.411522    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:22.709622 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:42:22.709536    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:42:24.079560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:24.079508    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:25.764331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:25.764284    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:28.001948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:28.001893    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:28.403284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:28.403248    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:42:29.822669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:29.822574    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:31.688709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:31.688669    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:33.706941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:33.706898    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:35.303044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:35.303005    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:36.527005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:36.526963    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:36.950365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:36.950330    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:37.298861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.298776    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"]
Nov 02 15:42:37.304322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.304298    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.323608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.323582    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"]
Nov 02 15:42:37.331800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.331772    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:42:37.331994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.331980    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:42:37.470447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.470411    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.470676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.470468    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4fz\" (UniqueName: \"kubernetes.io/projected/b292c8dc-7d92-4272-bc56-bb2839596653-kube-api-access-fv4fz\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.470763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.470695    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.571625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.571544    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4fz\" (UniqueName: \"kubernetes.io/projected/b292c8dc-7d92-4272-bc56-bb2839596653-kube-api-access-fv4fz\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.572030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.571684    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.572030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.571876    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.572339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.572311    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.574144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.574123    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.596156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.596128    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4fz\" (UniqueName: \"kubernetes.io/projected/b292c8dc-7d92-4272-bc56-bb2839596653-kube-api-access-fv4fz\") pod \"maas-api-revoke-sub-2ce097e9-lrjgm\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.620310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.620282    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:37.959135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:37.958843    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"]
Nov 02 15:42:38.623824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:38.622154    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:38.937516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:38.937483    7019 generic.go:356] "Generic (PLEG): container finished" podID="b292c8dc-7d92-4272-bc56-bb2839596653" containerID="01352cf4f815564b72a10785cc594795a32569e0925782dd3e68d0b9ca16e5e6" exitCode=0
Nov 02 15:42:38.937706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:38.937552    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm" event={"ID":"b292c8dc-7d92-4272-bc56-bb2839596653","Type":"ContainerDied","Data":"01352cf4f815564b72a10785cc594795a32569e0925782dd3e68d0b9ca16e5e6"}
Nov 02 15:42:38.937706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:38.937590    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm" event={"ID":"b292c8dc-7d92-4272-bc56-bb2839596653","Type":"ContainerStarted","Data":"ff403cb983eef67577ecf8af8d13020671c49bfb7c87eec0e12aaae67898777d"}
Nov 02 15:42:40.113465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.113439    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:40.209367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.209282    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b292c8dc-7d92-4272-bc56-bb2839596653-kube-api-access-fv4fz\" (UniqueName: \"kubernetes.io/projected/b292c8dc-7d92-4272-bc56-bb2839596653-kube-api-access-fv4fz\") pod \"b292c8dc-7d92-4272-bc56-bb2839596653\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") "
Nov 02 15:42:40.209537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.209398    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-xks-ca\") pod \"b292c8dc-7d92-4272-bc56-bb2839596653\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") "
Nov 02 15:42:40.209618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.209537    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-openshift-ca\") pod \"b292c8dc-7d92-4272-bc56-bb2839596653\" (UID: \"b292c8dc-7d92-4272-bc56-bb2839596653\") "
Nov 02 15:42:40.210000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.209968    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-openshift-ca" pod "b292c8dc-7d92-4272-bc56-bb2839596653" (UID: "b292c8dc-7d92-4272-bc56-bb2839596653"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:42:40.210245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.210228    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:42:40.212036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.212006    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-xks-ca" pod "b292c8dc-7d92-4272-bc56-bb2839596653" (UID: "b292c8dc-7d92-4272-bc56-bb2839596653"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:42:40.212136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.212093    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b292c8dc-7d92-4272-bc56-bb2839596653-kube-api-access-fv4fz" pod "b292c8dc-7d92-4272-bc56-bb2839596653" (UID: "b292c8dc-7d92-4272-bc56-bb2839596653"). InnerVolumeSpecName "kube-api-access-fv4fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:42:40.305614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.305562    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:40.316735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.312200    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv4fz\" (UniqueName: \"kubernetes.io/projected/b292c8dc-7d92-4272-bc56-bb2839596653-kube-api-access-fv4fz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:42:40.316735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.312240    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b292c8dc-7d92-4272-bc56-bb2839596653-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:42:40.955000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.954969    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"
Nov 02 15:42:40.955193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.955016    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm" event={"ID":"b292c8dc-7d92-4272-bc56-bb2839596653","Type":"ContainerDied","Data":"ff403cb983eef67577ecf8af8d13020671c49bfb7c87eec0e12aaae67898777d"}
Nov 02 15:42:40.955193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:40.955088    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff403cb983eef67577ecf8af8d13020671c49bfb7c87eec0e12aaae67898777d"
Nov 02 15:42:41.925898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:41.925865    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:43.492998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:43.492948    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:45.261044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:45.261007    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:46.477504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:46.477464    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:46.649961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:46.649926    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:47.627578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.627543    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:47.661822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.661786    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"]
Nov 02 15:42:47.662453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.662436    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b292c8dc-7d92-4272-bc56-bb2839596653" containerName="revoke-keys"
Nov 02 15:42:47.662502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.662454    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b292c8dc-7d92-4272-bc56-bb2839596653" containerName="revoke-keys"
Nov 02 15:42:47.662600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.662590    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b292c8dc-7d92-4272-bc56-bb2839596653" containerName="revoke-keys"
Nov 02 15:42:47.666300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.666278    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.682346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.682320    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:42:47.695805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.695778    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:42:47.701924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.701890    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"]
Nov 02 15:42:47.755266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.755232    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6zlp\" (UniqueName: \"kubernetes.io/projected/f3f665d0-8c4f-449c-8c84-d9d88fa03830-kube-api-access-r6zlp\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.755439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.755427    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.755483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.755467    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.856725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.856689    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.856929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.856743    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.856929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.856849    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6zlp\" (UniqueName: \"kubernetes.io/projected/f3f665d0-8c4f-449c-8c84-d9d88fa03830-kube-api-access-r6zlp\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.857395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.857356    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.859431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.859408    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.882070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.881994    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6zlp\" (UniqueName: \"kubernetes.io/projected/f3f665d0-8c4f-449c-8c84-d9d88fa03830-kube-api-access-r6zlp\") pod \"maas-api-revoke-sub-f4c25bf2-zbjjp\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:47.981238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:47.981205    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:48.298432 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:42:48.298397    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f665d0_8c4f_449c_8c84_d9d88fa03830.slice/crio-cc11952dfcb448e4a35be9db5c17e238f2efe714ac61c33b58574a6095bcc5c3 WatchSource:0}: Error finding container cc11952dfcb448e4a35be9db5c17e238f2efe714ac61c33b58574a6095bcc5c3: Status 404 returned error can't find the container with id cc11952dfcb448e4a35be9db5c17e238f2efe714ac61c33b58574a6095bcc5c3
Nov 02 15:42:48.299928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:48.299902    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"]
Nov 02 15:42:49.030033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.029995    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp" event={"ID":"f3f665d0-8c4f-449c-8c84-d9d88fa03830","Type":"ContainerStarted","Data":"2ea697898968508cd08d399ba43583efc9e49a0ec886a758f5e81feac4e675c6"}
Nov 02 15:42:49.030433 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.030055    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp" event={"ID":"f3f665d0-8c4f-449c-8c84-d9d88fa03830","Type":"ContainerStarted","Data":"cc11952dfcb448e4a35be9db5c17e238f2efe714ac61c33b58574a6095bcc5c3"}
Nov 02 15:42:49.075471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.075417    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp" podStartSLOduration=2.075402219 podStartE2EDuration="2.075402219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:49.071514137 +0000 UTC m=+2763.325467762" watchObservedRunningTime="2026-11-02 15:42:49.075402219 +0000 UTC m=+2763.329355879"
Nov 02 15:42:49.194594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.193637    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:49.422471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.422433    7019 scope.go:122] "RemoveContainer" containerID="05ff2f1c86957386246c0859b02c49b17fdbf1b70d0bf512acfa1eb65c8ce8e2"
Nov 02 15:42:49.432913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.432889    7019 scope.go:122] "RemoveContainer" containerID="568e8c55f2f906bd95b2ebe13913faf5016ed743d857763cd54990f877ec5d6e"
Nov 02 15:42:49.442241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.442223    7019 scope.go:122] "RemoveContainer" containerID="0d01551d7ad7e3845e6eee926719ca964d64053450942b89542bbc6cd8f64344"
Nov 02 15:42:49.451895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.451869    7019 scope.go:122] "RemoveContainer" containerID="eb63fc3bde74defc746201596ac3ab16cf71c4cb281c0a52a7cd1a4ca741aac0"
Nov 02 15:42:49.462173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.462154    7019 scope.go:122] "RemoveContainer" containerID="0a9cf682be23e64ddebbebfb70f6aca2d40bb2e2a5a56a65f4b0f6296622fb81"
Nov 02 15:42:49.473903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.473881    7019 scope.go:122] "RemoveContainer" containerID="0542ccaf1d4b56e408572e6afdcf0c6f2ff629140400c1682f982b66028984bb"
Nov 02 15:42:49.483407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.483389    7019 scope.go:122] "RemoveContainer" containerID="66915c0b04432e9074c42c961a2356a59bc901a5d57f8cb2ef63e9409c2dc7d9"
Nov 02 15:42:49.796699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:49.796593    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:50.039593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:50.039559    7019 generic.go:356] "Generic (PLEG): container finished" podID="f3f665d0-8c4f-449c-8c84-d9d88fa03830" containerID="2ea697898968508cd08d399ba43583efc9e49a0ec886a758f5e81feac4e675c6" exitCode=0
Nov 02 15:42:50.040068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:50.039676    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp" event={"ID":"f3f665d0-8c4f-449c-8c84-d9d88fa03830","Type":"ContainerDied","Data":"2ea697898968508cd08d399ba43583efc9e49a0ec886a758f5e81feac4e675c6"}
Nov 02 15:42:50.859950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:50.859914    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:51.017349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.017309    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:51.208223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.208191    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:51.327360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.327325    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-xks-ca\") pod \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") "
Nov 02 15:42:51.327627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.327611    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-openshift-ca\") pod \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") "
Nov 02 15:42:51.327762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.327682    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f665d0-8c4f-449c-8c84-d9d88fa03830-kube-api-access-r6zlp\" (UniqueName: \"kubernetes.io/projected/f3f665d0-8c4f-449c-8c84-d9d88fa03830-kube-api-access-r6zlp\") pod \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\" (UID: \"f3f665d0-8c4f-449c-8c84-d9d88fa03830\") "
Nov 02 15:42:51.327945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.327921    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-openshift-ca" pod "f3f665d0-8c4f-449c-8c84-d9d88fa03830" (UID: "f3f665d0-8c4f-449c-8c84-d9d88fa03830"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:42:51.328190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.328172    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:42:51.329505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.329481    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-xks-ca" pod "f3f665d0-8c4f-449c-8c84-d9d88fa03830" (UID: "f3f665d0-8c4f-449c-8c84-d9d88fa03830"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:42:51.329771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.329753    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f665d0-8c4f-449c-8c84-d9d88fa03830-kube-api-access-r6zlp" pod "f3f665d0-8c4f-449c-8c84-d9d88fa03830" (UID: "f3f665d0-8c4f-449c-8c84-d9d88fa03830"). InnerVolumeSpecName "kube-api-access-r6zlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:42:51.429962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.429874    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6zlp\" (UniqueName: \"kubernetes.io/projected/f3f665d0-8c4f-449c-8c84-d9d88fa03830-kube-api-access-r6zlp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:42:51.429962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:51.429907    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f665d0-8c4f-449c-8c84-d9d88fa03830-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:42:52.058452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:52.058421    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"
Nov 02 15:42:52.058639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:52.058449    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp" event={"ID":"f3f665d0-8c4f-449c-8c84-d9d88fa03830","Type":"ContainerDied","Data":"cc11952dfcb448e4a35be9db5c17e238f2efe714ac61c33b58574a6095bcc5c3"}
Nov 02 15:42:52.058639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:52.058513    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc11952dfcb448e4a35be9db5c17e238f2efe714ac61c33b58574a6095bcc5c3"
Nov 02 15:42:52.729745 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:42:52.729703    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:42:52.731180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:52.731152    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:54.300924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:54.300887    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:55.826889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:55.826851    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:57.375593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:57.375551    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:58.547778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:58.547738    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"]
Nov 02 15:42:58.603395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:58.603366    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:59.218394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:59.218357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:59.377443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:59.377411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:42:59.603744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:42:59.603558    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:00.100643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.100604    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:00.471889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.471853    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"]
Nov 02 15:43:00.473158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.473131    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f3f665d0-8c4f-449c-8c84-d9d88fa03830" containerName="revoke-keys"
Nov 02 15:43:00.473308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.473294    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f665d0-8c4f-449c-8c84-d9d88fa03830" containerName="revoke-keys"
Nov 02 15:43:00.473668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.473634    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f665d0-8c4f-449c-8c84-d9d88fa03830" containerName="revoke-keys"
Nov 02 15:43:00.489620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.489569    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.509533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.509372    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"]
Nov 02 15:43:00.517052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.516862    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5dd139-openshift-default-dockercfg-xzj4p\""
Nov 02 15:43:00.583035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.580097    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.625791    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-envoy\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.625852    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-podinfo\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.625914    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-credential-socket\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.626006    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-data\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.626107    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-socket\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.626147    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-certs\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.626287    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-token\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.626343    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4c5p\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-kube-api-access-q4c5p\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.633446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.626424    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istiod-ca-cert\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.727995    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-token\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728076    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4c5p\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-kube-api-access-q4c5p\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728156    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istiod-ca-cert\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728225    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-envoy\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728258    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-podinfo\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728317    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-credential-socket\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728406    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-data\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728516    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-socket\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.728725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.728565    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-certs\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.729392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.729236    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-certs\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.740674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.730411    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istiod-ca-cert\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.740674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.731069    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-credential-socket\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.740674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.731975    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-socket\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.740674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.732061    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-data\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.768670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.766955    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:00.782672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.774199    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-token\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.786664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.785149    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-podinfo\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.809672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.791695    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-envoy\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.847535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.847462    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4c5p\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-kube-api-access-q4c5p\") pod \"e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") " pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:00.999738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:00.993172    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:01.121625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:01.121585    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:01.462871 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:43:01.462829    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eedea98_57e6_4ca6_bdf0_39e3dc001705.slice/crio-609fb0292ff0c9672ed727c003d01b36e635f6940d62e39f9d9cfe9196495343 WatchSource:0}: Error finding container 609fb0292ff0c9672ed727c003d01b36e635f6940d62e39f9d9cfe9196495343: Status 404 returned error can't find the container with id 609fb0292ff0c9672ed727c003d01b36e635f6940d62e39f9d9cfe9196495343
Nov 02 15:43:01.463259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:01.463233    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"]
Nov 02 15:43:01.466487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:01.466446    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:43:01.466574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:01.466534    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:43:01.466612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:01.466576    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:43:02.149562 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:02.149524    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" event={"ID":"2eedea98-57e6-4ca6-bdf0-39e3dc001705","Type":"ContainerStarted","Data":"26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0"}
Nov 02 15:43:02.150009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:02.149582    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" event={"ID":"2eedea98-57e6-4ca6-bdf0-39e3dc001705","Type":"ContainerStarted","Data":"609fb0292ff0c9672ed727c003d01b36e635f6940d62e39f9d9cfe9196495343"}
Nov 02 15:43:02.709916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:02.709878    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:03.122361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:03.122324    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:03.128908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:03.128878    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:03.156493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:03.156466    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:03.157586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:03.157565    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:43:03.193699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:03.193628    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" podStartSLOduration=3.193608954 podStartE2EDuration="3.193608954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:43:02.199906053 +0000 UTC m=+2776.453859677" watchObservedRunningTime="2026-11-02 15:43:03.193608954 +0000 UTC m=+2777.447562579"
Nov 02 15:43:03.292328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:03.292288    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:04.487495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:04.487365    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:04.688536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:04.688500    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:04.859148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:04.859066    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:06.481816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.481752    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:06.599377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.599343    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"]
Nov 02 15:43:06.604350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.604317    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:06.624771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.624741    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"]
Nov 02 15:43:06.634232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.634207    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dd139\""
Nov 02 15:43:06.634390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.634327    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dd139-dockercfg-frn5t\""
Nov 02 15:43:06.674789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.674750    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:06.794723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.777329    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa31240-d342-494b-a284-125053d4769e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:06.794723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.777426    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wr5w\" (UniqueName: \"kubernetes.io/projected/baa31240-d342-494b-a284-125053d4769e-kube-api-access-4wr5w\") pod \"payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:06.887672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.884844    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa31240-d342-494b-a284-125053d4769e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:06.887672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.884899    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wr5w\" (UniqueName: \"kubernetes.io/projected/baa31240-d342-494b-a284-125053d4769e-kube-api-access-4wr5w\") pod \"payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:06.897670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.890670    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa31240-d342-494b-a284-125053d4769e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:06.914204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.914170    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wr5w\" (UniqueName: \"kubernetes.io/projected/baa31240-d342-494b-a284-125053d4769e-kube-api-access-4wr5w\") pod \"payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:06.923523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:06.923492    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:07.246131 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:43:07.246085    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa31240_d342_494b_a284_125053d4769e.slice/crio-a1c15512da31243069ecd1c1f70f5d6c16f3f99a2376ca3eba32fa8fda83e414 WatchSource:0}: Error finding container a1c15512da31243069ecd1c1f70f5d6c16f3f99a2376ca3eba32fa8fda83e414: Status 404 returned error can't find the container with id a1c15512da31243069ecd1c1f70f5d6c16f3f99a2376ca3eba32fa8fda83e414
Nov 02 15:43:07.246920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:07.246892    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"]
Nov 02 15:43:08.168248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:08.168209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:08.215875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:08.215827    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf" event={"ID":"baa31240-d342-494b-a284-125053d4769e","Type":"ContainerStarted","Data":"d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724"}
Nov 02 15:43:08.216099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:08.215898    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf" event={"ID":"baa31240-d342-494b-a284-125053d4769e","Type":"ContainerStarted","Data":"a1c15512da31243069ecd1c1f70f5d6c16f3f99a2376ca3eba32fa8fda83e414"}
Nov 02 15:43:08.216169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:08.216138    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:08.264206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:08.264154    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf" podStartSLOduration=2.264140259 podStartE2EDuration="2.264140259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:43:08.261004088 +0000 UTC m=+2782.514957713" watchObservedRunningTime="2026-11-02 15:43:08.264140259 +0000 UTC m=+2782.518093884"
Nov 02 15:43:08.344697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:08.344663    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:09.821242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:09.821208    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:11.704478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:11.704434    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:13.573051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:13.572985    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49" podUID="15379f67-4c39-4ab6-accc-ae1fe5dc679b" containerName="main" containerID="cri-o://35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23" gracePeriod=45
Nov 02 15:43:13.706093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:13.705983    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:13.894808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:13.894774    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:43:14.003792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.003753    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15379f67-4c39-4ab6-accc-ae1fe5dc679b-kube-api-access-swmmv\" (UniqueName: \"kubernetes.io/projected/15379f67-4c39-4ab6-accc-ae1fe5dc679b-kube-api-access-swmmv\") pod \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") "
Nov 02 15:43:14.004117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.003803    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-dshm\") pod \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") "
Nov 02 15:43:14.004117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.003895    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tmp-dir\") pod \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") "
Nov 02 15:43:14.004117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.003949    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-model-cache\") pod \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") "
Nov 02 15:43:14.004297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.004183    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-home\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-home\") pod \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") "
Nov 02 15:43:14.004297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.004215    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tls-certs\" (UniqueName: \"kubernetes.io/secret/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tls-certs\") pod \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\" (UID: \"15379f67-4c39-4ab6-accc-ae1fe5dc679b\") "
Nov 02 15:43:14.004297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.004242    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-model-cache" pod "15379f67-4c39-4ab6-accc-ae1fe5dc679b" (UID: "15379f67-4c39-4ab6-accc-ae1fe5dc679b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.004459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.004372    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tmp-dir" pod "15379f67-4c39-4ab6-accc-ae1fe5dc679b" (UID: "15379f67-4c39-4ab6-accc-ae1fe5dc679b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.004603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.004572    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-home" pod "15379f67-4c39-4ab6-accc-ae1fe5dc679b" (UID: "15379f67-4c39-4ab6-accc-ae1fe5dc679b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.005697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.004995    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.005697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.005017    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.005697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.005026    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.007016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.006989    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tls-certs" pod "15379f67-4c39-4ab6-accc-ae1fe5dc679b" (UID: "15379f67-4c39-4ab6-accc-ae1fe5dc679b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:43:14.007863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.007835    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15379f67-4c39-4ab6-accc-ae1fe5dc679b-kube-api-access-swmmv" pod "15379f67-4c39-4ab6-accc-ae1fe5dc679b" (UID: "15379f67-4c39-4ab6-accc-ae1fe5dc679b"). InnerVolumeSpecName "kube-api-access-swmmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:43:14.008506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.008074    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-dshm" pod "15379f67-4c39-4ab6-accc-ae1fe5dc679b" (UID: "15379f67-4c39-4ab6-accc-ae1fe5dc679b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:43:14.106242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.106203    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmmv\" (UniqueName: \"kubernetes.io/projected/15379f67-4c39-4ab6-accc-ae1fe5dc679b-kube-api-access-swmmv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.106242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.106239    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15379f67-4c39-4ab6-accc-ae1fe5dc679b-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.106242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.106251    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15379f67-4c39-4ab6-accc-ae1fe5dc679b-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:43:14.265078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.264981    7019 generic.go:356] "Generic (PLEG): container finished" podID="15379f67-4c39-4ab6-accc-ae1fe5dc679b" containerID="35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23" exitCode=0
Nov 02 15:43:14.265078 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.265061    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"
Nov 02 15:43:14.265300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.265059    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49" event={"ID":"15379f67-4c39-4ab6-accc-ae1fe5dc679b","Type":"ContainerDied","Data":"35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23"}
Nov 02 15:43:14.265300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.265190    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49" event={"ID":"15379f67-4c39-4ab6-accc-ae1fe5dc679b","Type":"ContainerDied","Data":"bdaabfb3d92aaaec1b7c870c56134db99af92a16f15bd5ebbf218b1d41d4c42f"}
Nov 02 15:43:14.265300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.265212    7019 scope.go:122] "RemoveContainer" containerID="35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23"
Nov 02 15:43:14.279436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.279401    7019 scope.go:122] "RemoveContainer" containerID="35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23"
Nov 02 15:43:14.279863 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:43:14.279842    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23\": container with ID starting with 35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23 not found: ID does not exist" containerID="35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23"
Nov 02 15:43:14.279954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.279872    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23"} err="failed to get container status \"35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23\": rpc error: code = NotFound desc = could not find container \"35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23\": container with ID starting with 35260b3d9061802f210e227a56223d7ba73d3005f3c6760e743c1691a5be9e23 not found: ID does not exist"
Nov 02 15:43:14.321916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.321878    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"]
Nov 02 15:43:14.329376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:14.329348    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-246137/rate-test-model-246137-kserve-769cddb97c-xrh49"]
Nov 02 15:43:15.310982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:15.310921    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:15.863270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:15.863239    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15379f67-4c39-4ab6-accc-ae1fe5dc679b" path="/var/lib/kubelet/pods/15379f67-4c39-4ab6-accc-ae1fe5dc679b/volumes"
Nov 02 15:43:17.136263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:17.136224    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:18.732219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:18.732183    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:19.232280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:19.232246    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:43:20.357086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:20.357050    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:22.003567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:22.001671    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:22.707517 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:43:22.707483    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:43:23.713994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:23.713952    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:25.374883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:25.373338    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:27.140502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:27.140408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:28.819239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:28.819202    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:30.449904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:30.449856    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:32.631843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:32.631804    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:34.314344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:34.314307    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:35.995947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:35.990131    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:37.614683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:37.613874    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:39.540183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:39.540133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:41.297952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:41.297890    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:42.989716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:42.989677    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:44.477176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:44.477141    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:44.651604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:44.651568    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:44.804139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:44.804048    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:46.560700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:46.560622    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:47.357959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:47.357918    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:47.534662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:47.534619    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:47.698354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:47.698315    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:48.338360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:48.335927    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:48.945679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:48.943415    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:43:49.548348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:49.548312    7019 scope.go:122] "RemoveContainer" containerID="d9a56ab69411d75ad8bba70567a08236e2d99162977209957da2729c78ecbcda"
Nov 02 15:43:49.560843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:49.560818    7019 scope.go:122] "RemoveContainer" containerID="0784eb140445bbac4227c2b43b7d726f27c3ab96ae53b9068b46acf279d0d78d"
Nov 02 15:43:49.571977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:49.571954    7019 scope.go:122] "RemoveContainer" containerID="dc9f5728bb67ca96eed86a51925ee38f7c87c67a52fdbe635958bb6d764b52b5"
Nov 02 15:43:49.583435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:49.583408    7019 scope.go:122] "RemoveContainer" containerID="437c54635cd4d9b6632f12b3700fefe5055b3765bc5be08ac1d309e543f40521"
Nov 02 15:43:50.027082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:50.027044    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:50.335766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:50.335660    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:50.505123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:50.505055    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:52.190903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:52.190869    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:52.714459 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:43:52.714413    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:43:54.126211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:54.122941    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:55.777813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:55.777776    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:57.520002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:57.519957    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:58.213664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:58.213616    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:43:59.330763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:43:59.330725    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:01.539420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:01.539379    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:03.343094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:03.343056    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:04.900071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:04.899974    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:05.172234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.172151    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6900-e10ef7"]
Nov 02 15:44:05.173177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.173157    7019 cpu_manager.go:414] "containerMap: removing container" podUID="15379f67-4c39-4ab6-accc-ae1fe5dc679b" containerName="main"
Nov 02 15:44:05.173238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.173180    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15379f67-4c39-4ab6-accc-ae1fe5dc679b" containerName="main"
Nov 02 15:44:05.173343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.173331    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="15379f67-4c39-4ab6-accc-ae1fe5dc679b" containerName="main"
Nov 02 15:44:05.179526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.179506    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:05.195555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.195522    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6900-e10ef7"]
Nov 02 15:44:05.203942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.203917    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dcqh6\""
Nov 02 15:44:05.302082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.302045    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92ld\" (UniqueName: \"kubernetes.io/projected/917baa9f-26ef-4234-8432-5bee131e4289-kube-api-access-m92ld\") pod \"test-curl-6900-e10ef7\" (UID: \"917baa9f-26ef-4234-8432-5bee131e4289\") " pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:05.403913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.403880    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92ld\" (UniqueName: \"kubernetes.io/projected/917baa9f-26ef-4234-8432-5bee131e4289-kube-api-access-m92ld\") pod \"test-curl-6900-e10ef7\" (UID: \"917baa9f-26ef-4234-8432-5bee131e4289\") " pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:05.441044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.440968    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92ld\" (UniqueName: \"kubernetes.io/projected/917baa9f-26ef-4234-8432-5bee131e4289-kube-api-access-m92ld\") pod \"test-curl-6900-e10ef7\" (UID: \"917baa9f-26ef-4234-8432-5bee131e4289\") " pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:05.501415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.501383    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:05.815028 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:44:05.814999    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod917baa9f_26ef_4234_8432_5bee131e4289.slice/crio-e93f223cfc9f531a442464c4fc2b10788eda513b235243f541f838dd8845d6a3 WatchSource:0}: Error finding container e93f223cfc9f531a442464c4fc2b10788eda513b235243f541f838dd8845d6a3: Status 404 returned error can't find the container with id e93f223cfc9f531a442464c4fc2b10788eda513b235243f541f838dd8845d6a3
Nov 02 15:44:05.817943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:05.817918    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6900-e10ef7"]
Nov 02 15:44:06.570448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:06.563790    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:06.765314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:06.765261    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6900-e10ef7" event={"ID":"917baa9f-26ef-4234-8432-5bee131e4289","Type":"ContainerStarted","Data":"b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188"}
Nov 02 15:44:06.765314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:06.765318    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6900-e10ef7" event={"ID":"917baa9f-26ef-4234-8432-5bee131e4289","Type":"ContainerStarted","Data":"e93f223cfc9f531a442464c4fc2b10788eda513b235243f541f838dd8845d6a3"}
Nov 02 15:44:06.806627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:06.806576    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6900-e10ef7" podStartSLOduration=1.806563929 podStartE2EDuration="1.806563929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:06.8054189 +0000 UTC m=+2841.059372724" watchObservedRunningTime="2026-11-02 15:44:06.806563929 +0000 UTC m=+2841.060517553"
Nov 02 15:44:07.749359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:07.749322    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6900-e10ef7"]
Nov 02 15:44:07.754849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:07.754819    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6900-e10ef7"]
Nov 02 15:44:07.987090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:07.987057    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"]
Nov 02 15:44:07.987799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:07.987781    7019 cpu_manager.go:414] "containerMap: removing container" podUID="917baa9f-26ef-4234-8432-5bee131e4289" containerName="test-curl-6900-e10ef7"
Nov 02 15:44:07.987874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:07.987799    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917baa9f-26ef-4234-8432-5bee131e4289" containerName="test-curl-6900-e10ef7"
Nov 02 15:44:07.987926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:07.987902    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="917baa9f-26ef-4234-8432-5bee131e4289" containerName="test-curl-6900-e10ef7"
Nov 02 15:44:07.991580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:07.991560    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.015697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.015615    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"]
Nov 02 15:44:08.023380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.023351    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:44:08.023536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.023466    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:44:08.144557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.144521    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.144769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.144576    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.144828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.144800    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjkd\" (UniqueName: \"kubernetes.io/projected/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-kube-api-access-spjkd\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.246678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.246607    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjkd\" (UniqueName: \"kubernetes.io/projected/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-kube-api-access-spjkd\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.246850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.246741    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.246850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.246778    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.247415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.247392    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.249167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.249143    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.272782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.272724    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjkd\" (UniqueName: \"kubernetes.io/projected/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-kube-api-access-spjkd\") pod \"maas-api-revoke-sub-6aa75d81-9j7pg\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.304608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.304571    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:08.306614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.306484    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:08.482558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.482520    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:08.644194 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:44:08.644159    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a0676e_8b8e_4b23_b0b8_6da21c75df2d.slice/crio-8bfe73f480626abcb3df4afb3738ea31c9f48539c4ad3aac4a6daf934c80b418 WatchSource:0}: Error finding container 8bfe73f480626abcb3df4afb3738ea31c9f48539c4ad3aac4a6daf934c80b418: Status 404 returned error can't find the container with id 8bfe73f480626abcb3df4afb3738ea31c9f48539c4ad3aac4a6daf934c80b418
Nov 02 15:44:08.644559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.644538    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"]
Nov 02 15:44:08.788982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.788946    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6900-e10ef7" podUID="917baa9f-26ef-4234-8432-5bee131e4289" containerName="test-curl-6900-e10ef7" containerID="cri-o://b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188" gracePeriod=2
Nov 02 15:44:08.789398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:08.789253    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg" event={"ID":"67a0676e-8b8e-4b23-b0b8-6da21c75df2d","Type":"ContainerStarted","Data":"8bfe73f480626abcb3df4afb3738ea31c9f48539c4ad3aac4a6daf934c80b418"}
Nov 02 15:44:09.801368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:09.801331    7019 generic.go:356] "Generic (PLEG): container finished" podID="67a0676e-8b8e-4b23-b0b8-6da21c75df2d" containerID="e922f00ce13e0ba69cbd270bbe9850c837ca3e00e09529329bcbde470680b13f" exitCode=0
Nov 02 15:44:09.801832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:09.801413    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg" event={"ID":"67a0676e-8b8e-4b23-b0b8-6da21c75df2d","Type":"ContainerDied","Data":"e922f00ce13e0ba69cbd270bbe9850c837ca3e00e09529329bcbde470680b13f"}
Nov 02 15:44:09.905375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:09.905341    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:11.003851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.003818    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:11.008225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.008195    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:11.097411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.097309    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-openshift-ca\") pod \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") "
Nov 02 15:44:11.097609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.097441    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-xks-ca\") pod \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") "
Nov 02 15:44:11.097609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.097539    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-kube-api-access-spjkd\" (UniqueName: \"kubernetes.io/projected/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-kube-api-access-spjkd\") pod \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\" (UID: \"67a0676e-8b8e-4b23-b0b8-6da21c75df2d\") "
Nov 02 15:44:11.097609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.097578    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917baa9f-26ef-4234-8432-5bee131e4289-kube-api-access-m92ld\" (UniqueName: \"kubernetes.io/projected/917baa9f-26ef-4234-8432-5bee131e4289-kube-api-access-m92ld\") pod \"917baa9f-26ef-4234-8432-5bee131e4289\" (UID: \"917baa9f-26ef-4234-8432-5bee131e4289\") "
Nov 02 15:44:11.097883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.097816    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-openshift-ca" pod "67a0676e-8b8e-4b23-b0b8-6da21c75df2d" (UID: "67a0676e-8b8e-4b23-b0b8-6da21c75df2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:44:11.098304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.098277    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:11.101369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.101336    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-xks-ca" pod "67a0676e-8b8e-4b23-b0b8-6da21c75df2d" (UID: "67a0676e-8b8e-4b23-b0b8-6da21c75df2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:44:11.101496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.101477    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-kube-api-access-spjkd" pod "67a0676e-8b8e-4b23-b0b8-6da21c75df2d" (UID: "67a0676e-8b8e-4b23-b0b8-6da21c75df2d"). InnerVolumeSpecName "kube-api-access-spjkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:44:11.101962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.101937    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917baa9f-26ef-4234-8432-5bee131e4289-kube-api-access-m92ld" pod "917baa9f-26ef-4234-8432-5bee131e4289" (UID: "917baa9f-26ef-4234-8432-5bee131e4289"). InnerVolumeSpecName "kube-api-access-m92ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:44:11.200076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.200032    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:11.200076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.200077    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spjkd\" (UniqueName: \"kubernetes.io/projected/67a0676e-8b8e-4b23-b0b8-6da21c75df2d-kube-api-access-spjkd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:11.200294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.200092    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m92ld\" (UniqueName: \"kubernetes.io/projected/917baa9f-26ef-4234-8432-5bee131e4289-kube-api-access-m92ld\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:11.245445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.245408    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"]
Nov 02 15:44:11.246114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.246098    7019 cpu_manager.go:414] "containerMap: removing container" podUID="67a0676e-8b8e-4b23-b0b8-6da21c75df2d" containerName="revoke-keys"
Nov 02 15:44:11.246161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.246116    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a0676e-8b8e-4b23-b0b8-6da21c75df2d" containerName="revoke-keys"
Nov 02 15:44:11.246224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.246214    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a0676e-8b8e-4b23-b0b8-6da21c75df2d" containerName="revoke-keys"
Nov 02 15:44:11.250020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.250004    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.273108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.273081    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"]
Nov 02 15:44:11.277294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.277273    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.279150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.279130    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e80c77-openshift-default-dockercfg-b7nkf\""
Nov 02 15:44:11.289846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.289818    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"]
Nov 02 15:44:11.314073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.314036    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"]
Nov 02 15:44:11.417941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.417898    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.417994    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418030    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418063    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418172    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cb29\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-kube-api-access-2cb29\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418202    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418221    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418341    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418367    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418388    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418472    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418492    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418520    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbmx\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-kube-api-access-bpbmx\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418623 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418610    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418632    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418665    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.418745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418727    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.418781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.418743    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.519733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519681    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.519733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519728    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.519973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519753    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.519973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519786    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.519973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519808    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.519973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519871    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.519973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519906    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.519973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519936    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.520314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.519981    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520045    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cb29\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-kube-api-access-2cb29\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.520314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520076    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520102    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.520314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520175    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520175    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.520314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520235    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520363    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520406    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520429    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520448    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520472    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520549    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520575    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520584    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520672    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbmx\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-kube-api-access-bpbmx\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.520752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520706    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.521320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.520874    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.521320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.521181    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.521320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.521261    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.522565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.522535    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.522686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.522662    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.522931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.522914    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.523164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.523147    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.560637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.560598    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.560959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.560933    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cb29\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-kube-api-access-2cb29\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.561004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.560937    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbmx\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-kube-api-access-bpbmx\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.565915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.565889    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.595949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.595901    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:11.752854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.752668    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:11.823007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.822967    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg" event={"ID":"67a0676e-8b8e-4b23-b0b8-6da21c75df2d","Type":"ContainerDied","Data":"8bfe73f480626abcb3df4afb3738ea31c9f48539c4ad3aac4a6daf934c80b418"}
Nov 02 15:44:11.823190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.823018    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"
Nov 02 15:44:11.823190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.823026    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bfe73f480626abcb3df4afb3738ea31c9f48539c4ad3aac4a6daf934c80b418"
Nov 02 15:44:11.826111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.826069    7019 generic.go:356] "Generic (PLEG): container finished" podID="917baa9f-26ef-4234-8432-5bee131e4289" containerID="b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188" exitCode=137
Nov 02 15:44:11.826266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.826127    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:11.826266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.826162    7019 scope.go:122] "RemoveContainer" containerID="b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188"
Nov 02 15:44:11.838032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.837999    7019 scope.go:122] "RemoveContainer" containerID="b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188"
Nov 02 15:44:11.838467 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:44:11.838402    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188\": container with ID starting with b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188 not found: ID does not exist" containerID="b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188"
Nov 02 15:44:11.838467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.838442    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188"} err="failed to get container status \"b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188\": rpc error: code = NotFound desc = could not find container \"b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188\": container with ID starting with b9d8fd6dcb10180cf26cdf5cc24e927a576aacdcf54a4b699fa9613a31708188 not found: ID does not exist"
Nov 02 15:44:11.863968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.863937    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917baa9f-26ef-4234-8432-5bee131e4289" path="/var/lib/kubelet/pods/917baa9f-26ef-4234-8432-5bee131e4289/volumes"
Nov 02 15:44:11.864182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.864141    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:11.880032 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:44:11.879980    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6900-e10ef7\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="917baa9f-26ef-4234-8432-5bee131e4289" pod="opendatahub/test-curl-6900-e10ef7"
Nov 02 15:44:11.941167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.941137    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"]
Nov 02 15:44:11.963000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.962952    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:11.963113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.963046    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:11.963113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:11.963086    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:12.188662 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:44:12.188616    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4512b064_bd7d_439c_8767_bf7d58c0f2fd.slice/crio-f473572d219144049708a79af2275850ac95db636fb8d2b4dea37a51668b43e6 WatchSource:0}: Error finding container f473572d219144049708a79af2275850ac95db636fb8d2b4dea37a51668b43e6: Status 404 returned error can't find the container with id f473572d219144049708a79af2275850ac95db636fb8d2b4dea37a51668b43e6
Nov 02 15:44:12.191420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.191393    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"]
Nov 02 15:44:12.192519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.192480    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:12.192597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.192567    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:12.192658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.192608    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:12.837508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.837474    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" event={"ID":"4512b064-bd7d-439c-8767-bf7d58c0f2fd","Type":"ContainerStarted","Data":"87533de51cef7af0570ab422e5c475b73926d5e705be3c312ecd59c4468a315a"}
Nov 02 15:44:12.837768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.837550    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" event={"ID":"4512b064-bd7d-439c-8767-bf7d58c0f2fd","Type":"ContainerStarted","Data":"f473572d219144049708a79af2275850ac95db636fb8d2b4dea37a51668b43e6"}
Nov 02 15:44:12.839324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.839296    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" event={"ID":"dfc1aa65-0e3b-48f6-9dee-756668d230f0","Type":"ContainerStarted","Data":"bf83d073a1bfb2514a2540e152340333e142af39557760a353b2026a20f7f6ed"}
Nov 02 15:44:12.839439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.839355    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" event={"ID":"dfc1aa65-0e3b-48f6-9dee-756668d230f0","Type":"ContainerStarted","Data":"ab0c9e7c667b54fdf5eaf3d33091247dffb0da8112df88a0b839d80396e07f31"}
Nov 02 15:44:12.864786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.864749    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:12.917514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.917450    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podStartSLOduration=1.9174278679999999 podStartE2EDuration="1.917427868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:12.910252168 +0000 UTC m=+2847.164205796" watchObservedRunningTime="2026-11-02 15:44:12.917427868 +0000 UTC m=+2847.171381493"
Nov 02 15:44:12.963805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:12.963753    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podStartSLOduration=1.9637358329999999 podStartE2EDuration="1.963735833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:12.96125967 +0000 UTC m=+2847.215213295" watchObservedRunningTime="2026-11-02 15:44:12.963735833 +0000 UTC m=+2847.217689459"
Nov 02 15:44:13.444364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:13.444323    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:13.606709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:13.603244    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:13.606709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:13.603445    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 15:44:13.606709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:13.603542    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 15:44:13.626693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:13.624067    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:13.876673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:13.875802    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 15:44:13.876673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:13.875865    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 15:44:14.616706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:14.613794    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 15:44:14.616706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:14.613864    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 15:44:14.892729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:14.881294    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 15:44:14.892729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:14.881358    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 15:44:15.325730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:15.323710    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:15.614670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:15.613085    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 15:44:15.614670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:15.613153    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 15:44:15.864728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:15.864602    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 15:44:15.864728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:15.864697    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 15:44:16.599571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:16.599071    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 15:44:16.599571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:16.599154    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 15:44:16.864625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:16.864589    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 15:44:16.886673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:16.876716    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 15:44:17.177735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:17.173785    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:17.599730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:17.599054    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 15:44:17.599730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:17.599132    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 15:44:17.873682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:17.864752    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 15:44:17.873682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:17.864814    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 15:44:18.443674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.441946    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"]
Nov 02 15:44:18.455163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.455129    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"]
Nov 02 15:44:18.455363 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.455246    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.537735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.536982    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:44:18.537910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.537807    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:44:18.555005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.554975    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qr7d\" (UniqueName: \"kubernetes.io/projected/98d6f4a7-4420-496e-96a1-996618c94d5e-kube-api-access-9qr7d\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.555165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.555014    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.555269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.555254    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.597134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.596983    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 15:44:18.597134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.597055    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 15:44:18.656538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.656508    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.657091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.657069    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qr7d\" (UniqueName: \"kubernetes.io/projected/98d6f4a7-4420-496e-96a1-996618c94d5e-kube-api-access-9qr7d\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.657246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.657234    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.658159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.658131    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.661681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.658992    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.718539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.718449    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qr7d\" (UniqueName: \"kubernetes.io/projected/98d6f4a7-4420-496e-96a1-996618c94d5e-kube-api-access-9qr7d\") pod \"maas-api-revoke-sub-01daa0fe-8hhfz\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.851640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.851531    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:18.865568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.865533    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 15:44:18.865850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.865820    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 15:44:19.002564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:18.948817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:19.013724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.012991    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"]
Nov 02 15:44:19.051028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.045579    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"]
Nov 02 15:44:19.051028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.045729    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.064743    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.064811    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.064851    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k42q\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-kube-api-access-5k42q\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.064896    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.064913    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.064935    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.065089    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.065115    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.075672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.065153    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.140576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.140545    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3805b8-openshift-default-dockercfg-h5w4z\""
Nov 02 15:44:19.171124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171089    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.171375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171159    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.171375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171213    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.171375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171266    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k42q\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-kube-api-access-5k42q\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.171375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171331    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.171375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171364    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171390    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171620    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171677    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171709    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171973    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.171993    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.172176    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.172439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.172407    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.174004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.173983    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.174004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.173998    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.190358 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:44:19.190321    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d6f4a7_4420_496e_96a1_996618c94d5e.slice/crio-267e20062cad40cb4cde7bfa879d3524a304363eec3ec11cdcb4300431f5320b WatchSource:0}: Error finding container 267e20062cad40cb4cde7bfa879d3524a304363eec3ec11cdcb4300431f5320b: Status 404 returned error can't find the container with id 267e20062cad40cb4cde7bfa879d3524a304363eec3ec11cdcb4300431f5320b
Nov 02 15:44:19.191190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.191168    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"]
Nov 02 15:44:19.198424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.198401    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.198513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.198497    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k42q\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-kube-api-access-5k42q\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.378925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.378897    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:19.603990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.603954    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:19.604473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.604448    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:19.605274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.605253    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:44:19.721694 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:44:19.720902    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc5a31d_14c9_4d57_acd3_e29a49a7f4c8.slice/crio-5807f6460f53bbac5d2956fca93f40203116bd2532d193b8d3d0fac68f48a5dd WatchSource:0}: Error finding container 5807f6460f53bbac5d2956fca93f40203116bd2532d193b8d3d0fac68f48a5dd: Status 404 returned error can't find the container with id 5807f6460f53bbac5d2956fca93f40203116bd2532d193b8d3d0fac68f48a5dd
Nov 02 15:44:19.724774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.724732    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:19.724952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.724824    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:19.724952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.724866    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:19.724952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.724918    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"]
Nov 02 15:44:19.871025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.870994    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:19.872163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.872137    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:19.872361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.872343    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:44:19.930221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.929538    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" event={"ID":"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8","Type":"ContainerStarted","Data":"48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943"}
Nov 02 15:44:19.930221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.929592    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" event={"ID":"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8","Type":"ContainerStarted","Data":"5807f6460f53bbac5d2956fca93f40203116bd2532d193b8d3d0fac68f48a5dd"}
Nov 02 15:44:19.932789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.932741    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz" event={"ID":"98d6f4a7-4420-496e-96a1-996618c94d5e","Type":"ContainerStarted","Data":"67d4f1f119112401d6524a5909f5091904e11842c978ed50133fbfbf322a7de1"}
Nov 02 15:44:19.932934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.932822    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz" event={"ID":"98d6f4a7-4420-496e-96a1-996618c94d5e","Type":"ContainerStarted","Data":"267e20062cad40cb4cde7bfa879d3524a304363eec3ec11cdcb4300431f5320b"}
Nov 02 15:44:19.990866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:19.990758    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podStartSLOduration=1.9907389580000001 podStartE2EDuration="1.990738958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:19.990618585 +0000 UTC m=+2854.244572209" watchObservedRunningTime="2026-11-02 15:44:19.990738958 +0000 UTC m=+2854.244692580"
Nov 02 15:44:20.100958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:20.100881    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz" podStartSLOduration=2.100861279 podStartE2EDuration="2.100861279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:20.095134007 +0000 UTC m=+2854.349087632" watchObservedRunningTime="2026-11-02 15:44:20.100861279 +0000 UTC m=+2854.354814904"
Nov 02 15:44:20.379844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:20.379803    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:20.381432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:20.381399    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 15:44:20.381572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:20.381462    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 15:44:20.918428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:20.918392    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:20.942012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:20.941978    7019 generic.go:356] "Generic (PLEG): container finished" podID="98d6f4a7-4420-496e-96a1-996618c94d5e" containerID="67d4f1f119112401d6524a5909f5091904e11842c978ed50133fbfbf322a7de1" exitCode=0
Nov 02 15:44:20.942179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:20.942066    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz" event={"ID":"98d6f4a7-4420-496e-96a1-996618c94d5e","Type":"ContainerDied","Data":"67d4f1f119112401d6524a5909f5091904e11842c978ed50133fbfbf322a7de1"}
Nov 02 15:44:21.123558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:21.118916    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"]
Nov 02 15:44:21.127215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:21.127143    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:21.133061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:21.133027    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c94c760d-4djb8"]
Nov 02 15:44:21.385672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:21.380837    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 15:44:21.385672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:21.380899    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 15:44:21.862817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:21.862778    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658e5f59-662f-4f04-8e70-44c375ccba55" path="/var/lib/kubelet/pods/658e5f59-662f-4f04-8e70-44c375ccba55/volumes"
Nov 02 15:44:22.210848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.210820    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:22.237132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.237103    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-openshift-ca\") pod \"98d6f4a7-4420-496e-96a1-996618c94d5e\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") "
Nov 02 15:44:22.237286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.237147    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-xks-ca\") pod \"98d6f4a7-4420-496e-96a1-996618c94d5e\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") "
Nov 02 15:44:22.237286 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.237252    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d6f4a7-4420-496e-96a1-996618c94d5e-kube-api-access-9qr7d\" (UniqueName: \"kubernetes.io/projected/98d6f4a7-4420-496e-96a1-996618c94d5e-kube-api-access-9qr7d\") pod \"98d6f4a7-4420-496e-96a1-996618c94d5e\" (UID: \"98d6f4a7-4420-496e-96a1-996618c94d5e\") "
Nov 02 15:44:22.237501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.237479    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-openshift-ca" pod "98d6f4a7-4420-496e-96a1-996618c94d5e" (UID: "98d6f4a7-4420-496e-96a1-996618c94d5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:44:22.237832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.237812    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:22.239333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.239310    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-xks-ca" pod "98d6f4a7-4420-496e-96a1-996618c94d5e" (UID: "98d6f4a7-4420-496e-96a1-996618c94d5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:44:22.239440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.239323    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d6f4a7-4420-496e-96a1-996618c94d5e-kube-api-access-9qr7d" pod "98d6f4a7-4420-496e-96a1-996618c94d5e" (UID: "98d6f4a7-4420-496e-96a1-996618c94d5e"). InnerVolumeSpecName "kube-api-access-9qr7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:44:22.339748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.339702    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d6f4a7-4420-496e-96a1-996618c94d5e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:22.339748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.339741    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qr7d\" (UniqueName: \"kubernetes.io/projected/98d6f4a7-4420-496e-96a1-996618c94d5e-kube-api-access-9qr7d\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:22.389670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.387360    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 15:44:22.389670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.387424    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 15:44:22.740459 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:44:22.740422    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:44:22.749238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:22.749204    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:23.026767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:23.025888    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:23.026767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:23.026166    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz" event={"ID":"98d6f4a7-4420-496e-96a1-996618c94d5e","Type":"ContainerDied","Data":"267e20062cad40cb4cde7bfa879d3524a304363eec3ec11cdcb4300431f5320b"}
Nov 02 15:44:23.026767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:23.026202    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="267e20062cad40cb4cde7bfa879d3524a304363eec3ec11cdcb4300431f5320b"
Nov 02 15:44:23.026767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:23.026254    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"
Nov 02 15:44:23.385672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:23.383045    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 15:44:23.385672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:23.383111    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 15:44:24.384672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:24.384037    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 15:44:24.384672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:24.384101    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 15:44:24.696404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:24.696232    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:25.384672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:25.383153    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 15:44:25.384672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:25.383226    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 15:44:26.061955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:26.061908    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:26.243965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:26.243926    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:26.466756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:26.466637    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:26.467545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:26.467508    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:26.469246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:26.469220    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:44:27.252575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:27.252531    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:29.184699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:29.184219    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:30.826416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:30.826313    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:31.024268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:31.024228    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:31.052754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:31.052719    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"]
Nov 02 15:44:31.058785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:31.058750    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12780cf8-qr4qp"]
Nov 02 15:44:31.862698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:31.862666    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affc85ff-e4d5-466a-983f-79e25be736d8" path="/var/lib/kubelet/pods/affc85ff-e4d5-466a-983f-79e25be736d8/volumes"
Nov 02 15:44:32.716587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:32.716538    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:34.418038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:34.418003    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:35.681565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:35.681527    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:35.886754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:35.886720    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:36.845062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:36.845028    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:37.191344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:37.191269    7019 generic.go:356] "Generic (PLEG): container finished" podID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerID="26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0" exitCode=0
Nov 02 15:44:37.193032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:37.191349    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" event={"ID":"2eedea98-57e6-4ca6-bdf0-39e3dc001705","Type":"ContainerDied","Data":"26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0"}
Nov 02 15:44:37.193032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:37.192135    7019 scope.go:122] "RemoveContainer" containerID="26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0"
Nov 02 15:44:37.199388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:37.199344    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:37.199520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:37.199448    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:37.199584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:37.199506    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:38.208015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:38.207979    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" event={"ID":"2eedea98-57e6-4ca6-bdf0-39e3dc001705","Type":"ContainerStarted","Data":"272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1"}
Nov 02 15:44:38.612299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:38.612259    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:39.122699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:39.122634    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:44:39.122877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:39.122835    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:44:39.122940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:39.122877    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 02 15:44:40.122518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:40.122486    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:44:40.122945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:40.122559    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 02 15:44:40.332317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:40.332278    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:41.122225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:41.122187    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:44:41.122427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:41.122255    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 02 15:44:42.057670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.049829    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:42.123206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.122283    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:44:42.123206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.122367    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 02 15:44:42.823658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.823608    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"]
Nov 02 15:44:42.824335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.824307    7019 cpu_manager.go:414] "containerMap: removing container" podUID="98d6f4a7-4420-496e-96a1-996618c94d5e" containerName="revoke-keys"
Nov 02 15:44:42.824335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.824327    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d6f4a7-4420-496e-96a1-996618c94d5e" containerName="revoke-keys"
Nov 02 15:44:42.824466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.824435    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d6f4a7-4420-496e-96a1-996618c94d5e" containerName="revoke-keys"
Nov 02 15:44:42.832103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.832072    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:42.849015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.848983    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"]
Nov 02 15:44:42.858437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.858411    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:44:42.858586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.858538    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:44:42.907861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.907821    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzsl\" (UniqueName: \"kubernetes.io/projected/a5437a45-014a-473a-9178-227e9a494372-kube-api-access-vlzsl\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:42.908082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.908025    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5437a45-014a-473a-9178-227e9a494372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:42.908187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:42.908170    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5437a45-014a-473a-9178-227e9a494372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.009859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.009295    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzsl\" (UniqueName: \"kubernetes.io/projected/a5437a45-014a-473a-9178-227e9a494372-kube-api-access-vlzsl\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.009859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.009775    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5437a45-014a-473a-9178-227e9a494372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.010133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.010009    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5437a45-014a-473a-9178-227e9a494372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.010812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.010782    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5437a45-014a-473a-9178-227e9a494372-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.012413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.012384    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5437a45-014a-473a-9178-227e9a494372-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.037623 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.037592    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzsl\" (UniqueName: \"kubernetes.io/projected/a5437a45-014a-473a-9178-227e9a494372-kube-api-access-vlzsl\") pod \"maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.122202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.122169    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 02 15:44:43.122602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.122222    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 02 15:44:43.156382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.156353    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:43.527324 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:44:43.527277    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5437a45_014a_473a_9178_227e9a494372.slice/crio-d1bb8d2af10c6145785e1bcd1829542e88782b020d92005004dd6181def3bc43 WatchSource:0}: Error finding container d1bb8d2af10c6145785e1bcd1829542e88782b020d92005004dd6181def3bc43: Status 404 returned error can't find the container with id d1bb8d2af10c6145785e1bcd1829542e88782b020d92005004dd6181def3bc43
Nov 02 15:44:43.527625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.527599    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"]
Nov 02 15:44:43.862259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:43.862191    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:44.127487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:44.127457    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:44:44.128015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:44.127874    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:44:44.128507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:44.128487    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:44:44.258984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:44.258941    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl" event={"ID":"a5437a45-014a-473a-9178-227e9a494372","Type":"ContainerStarted","Data":"628a8ab5d8786f5ac8c6adaab3a1004efaea115516829bb1d6d0337ed9b95393"}
Nov 02 15:44:44.259179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:44.259013    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl" event={"ID":"a5437a45-014a-473a-9178-227e9a494372","Type":"ContainerStarted","Data":"d1bb8d2af10c6145785e1bcd1829542e88782b020d92005004dd6181def3bc43"}
Nov 02 15:44:45.268187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:45.268142    7019 generic.go:356] "Generic (PLEG): container finished" podID="a5437a45-014a-473a-9178-227e9a494372" containerID="628a8ab5d8786f5ac8c6adaab3a1004efaea115516829bb1d6d0337ed9b95393" exitCode=0
Nov 02 15:44:45.268635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:45.268228    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl" event={"ID":"a5437a45-014a-473a-9178-227e9a494372","Type":"ContainerDied","Data":"628a8ab5d8786f5ac8c6adaab3a1004efaea115516829bb1d6d0337ed9b95393"}
Nov 02 15:44:45.627370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:45.627324    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:46.447246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.447221    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:46.572315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.572230    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5437a45-014a-473a-9178-227e9a494372-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5437a45-014a-473a-9178-227e9a494372-maas-api-openshift-ca\") pod \"a5437a45-014a-473a-9178-227e9a494372\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") "
Nov 02 15:44:46.572470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.572415    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5437a45-014a-473a-9178-227e9a494372-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5437a45-014a-473a-9178-227e9a494372-maas-api-xks-ca\") pod \"a5437a45-014a-473a-9178-227e9a494372\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") "
Nov 02 15:44:46.572541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.572467    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5437a45-014a-473a-9178-227e9a494372-kube-api-access-vlzsl\" (UniqueName: \"kubernetes.io/projected/a5437a45-014a-473a-9178-227e9a494372-kube-api-access-vlzsl\") pod \"a5437a45-014a-473a-9178-227e9a494372\" (UID: \"a5437a45-014a-473a-9178-227e9a494372\") "
Nov 02 15:44:46.572722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.572691    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5437a45-014a-473a-9178-227e9a494372-maas-api-openshift-ca" pod "a5437a45-014a-473a-9178-227e9a494372" (UID: "a5437a45-014a-473a-9178-227e9a494372"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:44:46.573201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.573175    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5437a45-014a-473a-9178-227e9a494372-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.574773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.574741    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5437a45-014a-473a-9178-227e9a494372-maas-api-xks-ca" pod "a5437a45-014a-473a-9178-227e9a494372" (UID: "a5437a45-014a-473a-9178-227e9a494372"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:44:46.575392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.575363    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5437a45-014a-473a-9178-227e9a494372-kube-api-access-vlzsl" pod "a5437a45-014a-473a-9178-227e9a494372" (UID: "a5437a45-014a-473a-9178-227e9a494372"). InnerVolumeSpecName "kube-api-access-vlzsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:44:46.606905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.606875    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"]
Nov 02 15:44:46.675056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.675018    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5437a45-014a-473a-9178-227e9a494372-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.675056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.675050    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzsl\" (UniqueName: \"kubernetes.io/projected/a5437a45-014a-473a-9178-227e9a494372-kube-api-access-vlzsl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:46.810614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.810572    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:46.979449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:46.979409    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:47.287549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:47.287459    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"
Nov 02 15:44:47.287549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:47.287484    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl" event={"ID":"a5437a45-014a-473a-9178-227e9a494372","Type":"ContainerDied","Data":"d1bb8d2af10c6145785e1bcd1829542e88782b020d92005004dd6181def3bc43"}
Nov 02 15:44:47.287549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:47.287544    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1bb8d2af10c6145785e1bcd1829542e88782b020d92005004dd6181def3bc43"
Nov 02 15:44:47.345949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:47.345915    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:49.184278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:49.184246    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:50.065216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:50.065128    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:51.358788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.358737    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:51.607227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.607176    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf" podUID="baa31240-d342-494b-a284-125053d4769e" containerName="payload-processing" containerID="cri-o://d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724" gracePeriod=25
Nov 02 15:44:51.905507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.905479    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:44:51.969577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.969542    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa31240-d342-494b-a284-125053d4769e-kube-api-access-4wr5w\" (UniqueName: \"kubernetes.io/projected/baa31240-d342-494b-a284-125053d4769e-kube-api-access-4wr5w\") pod \"baa31240-d342-494b-a284-125053d4769e\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") "
Nov 02 15:44:51.969786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.969747    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa31240-d342-494b-a284-125053d4769e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa31240-d342-494b-a284-125053d4769e-plugins-config-volume\") pod \"baa31240-d342-494b-a284-125053d4769e\" (UID: \"baa31240-d342-494b-a284-125053d4769e\") "
Nov 02 15:44:51.970329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.970269    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa31240-d342-494b-a284-125053d4769e-plugins-config-volume" pod "baa31240-d342-494b-a284-125053d4769e" (UID: "baa31240-d342-494b-a284-125053d4769e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:44:51.970507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.970485    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa31240-d342-494b-a284-125053d4769e-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:51.972316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:51.972292    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa31240-d342-494b-a284-125053d4769e-kube-api-access-4wr5w" pod "baa31240-d342-494b-a284-125053d4769e" (UID: "baa31240-d342-494b-a284-125053d4769e"). InnerVolumeSpecName "kube-api-access-4wr5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:44:52.071509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.071470    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wr5w\" (UniqueName: \"kubernetes.io/projected/baa31240-d342-494b-a284-125053d4769e-kube-api-access-4wr5w\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:44:52.332236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.332141    7019 generic.go:356] "Generic (PLEG): container finished" podID="baa31240-d342-494b-a284-125053d4769e" containerID="d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724" exitCode=0
Nov 02 15:44:52.332236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.332196    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"
Nov 02 15:44:52.332432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.332237    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf" event={"ID":"baa31240-d342-494b-a284-125053d4769e","Type":"ContainerDied","Data":"d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724"}
Nov 02 15:44:52.332432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.332299    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf" event={"ID":"baa31240-d342-494b-a284-125053d4769e","Type":"ContainerDied","Data":"a1c15512da31243069ecd1c1f70f5d6c16f3f99a2376ca3eba32fa8fda83e414"}
Nov 02 15:44:52.332432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.332328    7019 scope.go:122] "RemoveContainer" containerID="d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724"
Nov 02 15:44:52.344915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.344891    7019 scope.go:122] "RemoveContainer" containerID="d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724"
Nov 02 15:44:52.345167 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:44:52.345146    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724\": container with ID starting with d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724 not found: ID does not exist" containerID="d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724"
Nov 02 15:44:52.345224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.345178    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724"} err="failed to get container status \"d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724\": rpc error: code = NotFound desc = could not find container \"d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724\": container with ID starting with d4491fd64a59d9dabcb736813a4b62d783826ad2b187bbb215df7494387ba724 not found: ID does not exist"
Nov 02 15:44:52.395630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.395592    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"]
Nov 02 15:44:52.408224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:52.408193    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dd139-fbc9cc5bf-xzlwf"]
Nov 02 15:44:52.706328 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:44:52.706289    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:44:53.030132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:53.030039    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:53.862888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:53.862857    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa31240-d342-494b-a284-125053d4769e" path="/var/lib/kubelet/pods/baa31240-d342-494b-a284-125053d4769e/volumes"
Nov 02 15:44:54.037865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:54.037834    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"]
Nov 02 15:44:54.051896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:54.051861    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89747308-rg2lt"]
Nov 02 15:44:54.055140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:54.055115    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"]
Nov 02 15:44:54.058192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:54.058170    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac81dddb-hhhsq"]
Nov 02 15:44:54.610312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:54.610277    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:55.876175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:55.876132    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01edccac-720b-4ee2-b39b-e54287931f64" path="/var/lib/kubelet/pods/01edccac-720b-4ee2-b39b-e54287931f64/volumes"
Nov 02 15:44:55.876817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:55.876795    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64b0db9e-7ac6-48ed-901a-6b817dbd3239" path="/var/lib/kubelet/pods/64b0db9e-7ac6-48ed-901a-6b817dbd3239/volumes"
Nov 02 15:44:56.338197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.338059    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:56.344614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.344322    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"]
Nov 02 15:44:56.345991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.345954    7019 cpu_manager.go:414] "containerMap: removing container" podUID="baa31240-d342-494b-a284-125053d4769e" containerName="payload-processing"
Nov 02 15:44:56.345991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.345991    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa31240-d342-494b-a284-125053d4769e" containerName="payload-processing"
Nov 02 15:44:56.346209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.346066    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a5437a45-014a-473a-9178-227e9a494372" containerName="revoke-keys"
Nov 02 15:44:56.346209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.346081    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5437a45-014a-473a-9178-227e9a494372" containerName="revoke-keys"
Nov 02 15:44:56.346324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.346297    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa31240-d342-494b-a284-125053d4769e" containerName="payload-processing"
Nov 02 15:44:56.346324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.346323    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5437a45-014a-473a-9178-227e9a494372" containerName="revoke-keys"
Nov 02 15:44:56.351611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.351589    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.400305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.400267    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"]
Nov 02 15:44:56.438547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.438507    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a330d41a-4746-4900-af0d-2b9917fb199d-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.438767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.438690    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.438930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.438852    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a330d41a-4746-4900-af0d-2b9917fb199d-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.438930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.438903    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlvf\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-kube-api-access-5mlvf\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.439089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.439035    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.439089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.439077    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.439212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.439129    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.439212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.439189    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.439212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.439206    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540259    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a330d41a-4746-4900-af0d-2b9917fb199d-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540318    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlvf\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-kube-api-access-5mlvf\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540371    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540421    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540454    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540514    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540623    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540759    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a330d41a-4746-4900-af0d-2b9917fb199d-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540845    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.540924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540879    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.541046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.540950    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.541192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.541170    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.541296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.541221    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.541566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.541542    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a330d41a-4746-4900-af0d-2b9917fb199d-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.543040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.543011    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a330d41a-4746-4900-af0d-2b9917fb199d-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.543150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.543068    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.568499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.568471    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.568750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.568725    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlvf\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-kube-api-access-5mlvf\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:56.671231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:56.671199    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:57.006044 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:44:57.005182    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda330d41a_4746_4900_af0d_2b9917fb199d.slice/crio-0453821cc928b06624eea8198782485da6c7783d796348067c80814e4885b117 WatchSource:0}: Error finding container 0453821cc928b06624eea8198782485da6c7783d796348067c80814e4885b117: Status 404 returned error can't find the container with id 0453821cc928b06624eea8198782485da6c7783d796348067c80814e4885b117
Nov 02 15:44:57.006547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.006052    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"]
Nov 02 15:44:57.009105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.009061    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:57.009212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.009150    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:57.009212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.009196    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:44:57.378726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.378681    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" event={"ID":"a330d41a-4746-4900-af0d-2b9917fb199d","Type":"ContainerStarted","Data":"601e39cdea53e8623e160c53c487fa70d91369f2a2f999d10d047b46c089cd52"}
Nov 02 15:44:57.378938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.378755    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" event={"ID":"a330d41a-4746-4900-af0d-2b9917fb199d","Type":"ContainerStarted","Data":"0453821cc928b06624eea8198782485da6c7783d796348067c80814e4885b117"}
Nov 02 15:44:57.432750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.432696    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podStartSLOduration=1.432680288 podStartE2EDuration="1.432680288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:44:57.430525519 +0000 UTC m=+2891.684479145" watchObservedRunningTime="2026-11-02 15:44:57.432680288 +0000 UTC m=+2891.686633909"
Nov 02 15:44:57.672022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:57.671935    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:44:58.049176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:58.049042    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:44:58.674689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:58.674454    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 15:44:58.674689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:58.674534    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 15:44:59.017163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:59.017078    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:44:59.677673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:59.673988    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 15:44:59.677673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:59.674049    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 15:44:59.959510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:44:59.959415    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:00.196674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.193020    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"]
Nov 02 15:45:00.199114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.199090    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:00.200380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.200355    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"]
Nov 02 15:45:00.205481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.205452    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:00.207146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.207123    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"]
Nov 02 15:45:00.251835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.220196    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"]
Nov 02 15:45:00.251835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.221952    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:00.251835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.230912    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.251835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.244672    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"]
Nov 02 15:45:00.377445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.342379    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"]
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418052    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c120c3-da25-4a39-8a28-916dce067311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418116    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxm22\" (UniqueName: \"kubernetes.io/projected/75fa808b-30f1-4090-859a-c8bb8fd9a999-kube-api-access-mxm22\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418228    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98247b5f-4631-4cbe-9fc7-a447913966b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418349    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b568d3ce-7e0d-4909-8835-f2efc6bd4793-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418386    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46q4k\" (UniqueName: \"kubernetes.io/projected/e4c120c3-da25-4a39-8a28-916dce067311-kube-api-access-46q4k\") pod \"maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418489    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmcj\" (UniqueName: \"kubernetes.io/projected/b568d3ce-7e0d-4909-8835-f2efc6bd4793-kube-api-access-sbmcj\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418548    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tx5g\" (UniqueName: \"kubernetes.io/projected/98247b5f-4631-4cbe-9fc7-a447913966b7-kube-api-access-7tx5g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:00.419053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.418574    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75fa808b-30f1-4090-859a-c8bb8fd9a999-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:00.429915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.429088    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521347    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b568d3ce-7e0d-4909-8835-f2efc6bd4793-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521410    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46q4k\" (UniqueName: \"kubernetes.io/projected/e4c120c3-da25-4a39-8a28-916dce067311-kube-api-access-46q4k\") pod \"maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521526    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmcj\" (UniqueName: \"kubernetes.io/projected/b568d3ce-7e0d-4909-8835-f2efc6bd4793-kube-api-access-sbmcj\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521586    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tx5g\" (UniqueName: \"kubernetes.io/projected/98247b5f-4631-4cbe-9fc7-a447913966b7-kube-api-access-7tx5g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521614    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75fa808b-30f1-4090-859a-c8bb8fd9a999-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521739    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c120c3-da25-4a39-8a28-916dce067311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521777    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxm22\" (UniqueName: \"kubernetes.io/projected/75fa808b-30f1-4090-859a-c8bb8fd9a999-kube-api-access-mxm22\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:00.523742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.521899    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98247b5f-4631-4cbe-9fc7-a447913966b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:00.524347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.523930    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b568d3ce-7e0d-4909-8835-f2efc6bd4793-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:00.545688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.524737    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c120c3-da25-4a39-8a28-916dce067311-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.545688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.525772    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98247b5f-4631-4cbe-9fc7-a447913966b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:00.545688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.531944    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75fa808b-30f1-4090-859a-c8bb8fd9a999-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:00.554878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.551870    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"]
Nov 02 15:45:00.577862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.575183    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"]
Nov 02 15:45:00.601891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.601852    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"]
Nov 02 15:45:00.749529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.739453    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46q4k\" (UniqueName: \"kubernetes.io/projected/e4c120c3-da25-4a39-8a28-916dce067311-kube-api-access-46q4k\") pod \"maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.829740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.820329    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 15:45:00.829740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.820390    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 15:45:00.853828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.845071    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:00.853828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.852190    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"]
Nov 02 15:45:00.879969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.874306    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:00.948663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.948595    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxm22\" (UniqueName: \"kubernetes.io/projected/75fa808b-30f1-4090-859a-c8bb8fd9a999-kube-api-access-mxm22\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:00.948868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.948737    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvwz\" (UniqueName: \"kubernetes.io/projected/fa7b87a4-682a-4462-9610-63ee828259e3-kube-api-access-grvwz\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:00.948868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.948793    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b87a4-682a-4462-9610-63ee828259e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:00.958425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:00.958390    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tx5g\" (UniqueName: \"kubernetes.io/projected/98247b5f-4631-4cbe-9fc7-a447913966b7-kube-api-access-7tx5g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:01.154669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.151818    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmcj\" (UniqueName: \"kubernetes.io/projected/b568d3ce-7e0d-4909-8835-f2efc6bd4793-kube-api-access-sbmcj\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:01.204069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.204026    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:01.470191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.459873    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:01.538228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.489114    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvwz\" (UniqueName: \"kubernetes.io/projected/fa7b87a4-682a-4462-9610-63ee828259e3-kube-api-access-grvwz\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:01.582572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.522010    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:01.582572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.536751    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:01.630323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.600458    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"]
Nov 02 15:45:01.630323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.600572    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"]
Nov 02 15:45:01.658981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.648880    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:01.740730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.726114    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 15:45:01.740730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.726170    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 15:45:01.758695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.750432    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:01.758695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.755837    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvwz\" (UniqueName: \"kubernetes.io/projected/fa7b87a4-682a-4462-9610-63ee828259e3-kube-api-access-grvwz\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:01.758695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.758149    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b87a4-682a-4462-9610-63ee828259e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:01.775677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.769598    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b87a4-682a-4462-9610-63ee828259e3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:01.803875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.803290    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:01.898437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.887192    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmbc\" (UniqueName: \"kubernetes.io/projected/08cc86fb-8725-48e1-9a73-7ad63bc05755-kube-api-access-zpmbc\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:01.898437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.887264    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cc86fb-8725-48e1-9a73-7ad63bc05755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:01.898437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.887783    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a625702-a861-4efb-9bf5-b50c4f3119cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:01.898437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:01.887817    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdcjt\" (UniqueName: \"kubernetes.io/projected/1a625702-a861-4efb-9bf5-b50c4f3119cf-kube-api-access-bdcjt\") pod \"maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:02.040558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.028750    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.139136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.044985    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a625702-a861-4efb-9bf5-b50c4f3119cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:02.139136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.045036    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdcjt\" (UniqueName: \"kubernetes.io/projected/1a625702-a861-4efb-9bf5-b50c4f3119cf-kube-api-access-bdcjt\") pod \"maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:02.139136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.045097    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmbc\" (UniqueName: \"kubernetes.io/projected/08cc86fb-8725-48e1-9a73-7ad63bc05755-kube-api-access-zpmbc\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:02.139136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.045133    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cc86fb-8725-48e1-9a73-7ad63bc05755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:02.139136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.046055    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cc86fb-8725-48e1-9a73-7ad63bc05755-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:02.139136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.048565    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a625702-a861-4efb-9bf5-b50c4f3119cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:02.263119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.230321    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdcjt\" (UniqueName: \"kubernetes.io/projected/1a625702-a861-4efb-9bf5-b50c4f3119cf-kube-api-access-bdcjt\") pod \"maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:02.315778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.306192    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:02.349888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.349805    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.352551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.351462    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.384053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.380395    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm" podUID="32f7da44-9c7b-42fc-8b24-89f61b2df9e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.216:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.450910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.448670    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmbc\" (UniqueName: \"kubernetes.io/projected/08cc86fb-8725-48e1-9a73-7ad63bc05755-kube-api-access-zpmbc\") pod \"maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:02.485716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.456450    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"]
Nov 02 15:45:02.485716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.458924    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:02.486674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.486185    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.529973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.524835    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"]
Nov 02 15:45:02.533790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.533760    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:02.538549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.538516    7019 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.641501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.641459    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2" podUID="3b712526-9385-4c26-b37e-8b0bee370de1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.219:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.691354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.691308    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:02.698705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.698670    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"]
Nov 02 15:45:02.698971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.698954    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"]
Nov 02 15:45:02.716182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.716103    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:02.716978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.716946    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 15:45:02.717189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.717163    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 15:45:02.717396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.717362    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935acbcf-5b16-4096-8a48-73579d2b6505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:02.717705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.717680    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdr9l\" (UniqueName: \"kubernetes.io/projected/73bca953-9136-42bd-916d-1e3f1915b53a-kube-api-access-tdr9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:02.717871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.717852    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2r6\" (UniqueName: \"kubernetes.io/projected/935acbcf-5b16-4096-8a48-73579d2b6505-kube-api-access-ht2r6\") pod \"maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:02.718212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.718189    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bca953-9136-42bd-916d-1e3f1915b53a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:02.728756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.725880    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:02.757672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.753436    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 15:45:02.798674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.772348    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"]
Nov 02 15:45:02.839695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.819444    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:02.839695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.819513    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935acbcf-5b16-4096-8a48-73579d2b6505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:02.839695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.819605    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdr9l\" (UniqueName: \"kubernetes.io/projected/73bca953-9136-42bd-916d-1e3f1915b53a-kube-api-access-tdr9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:02.839695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.819639    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2r6\" (UniqueName: \"kubernetes.io/projected/935acbcf-5b16-4096-8a48-73579d2b6505-kube-api-access-ht2r6\") pod \"maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:02.839695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.819767    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfdd\" (UniqueName: \"kubernetes.io/projected/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-kube-api-access-vdfdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:02.839695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.819886    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bca953-9136-42bd-916d-1e3f1915b53a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:02.863839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.858664    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:45:02.863839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.860103    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935acbcf-5b16-4096-8a48-73579d2b6505-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:02.868684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.865004    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bca953-9136-42bd-916d-1e3f1915b53a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:02.900140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.899474    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2r6\" (UniqueName: \"kubernetes.io/projected/935acbcf-5b16-4096-8a48-73579d2b6505-kube-api-access-ht2r6\") pod \"maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:02.927276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.922194    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdr9l\" (UniqueName: \"kubernetes.io/projected/73bca953-9136-42bd-916d-1e3f1915b53a-kube-api-access-tdr9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:02.927276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.923525    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:02.927276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.923762    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfdd\" (UniqueName: \"kubernetes.io/projected/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-kube-api-access-vdfdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:02.927276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:02.924783    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:03.045263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.045226    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"]
Nov 02 15:45:03.055681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.052590    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfdd\" (UniqueName: \"kubernetes.io/projected/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-kube-api-access-vdfdd\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:03.108817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.108496    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:03.111815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.109296    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:03.133834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.133794    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"]
Nov 02 15:45:03.134079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.134053    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"]
Nov 02 15:45:03.134348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.134333    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.147254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.147113    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"]
Nov 02 15:45:03.155423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.155400    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:03.214692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.214017    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"]
Nov 02 15:45:03.218835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.217883    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.232746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.232277    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"]
Nov 02 15:45:03.232746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.232341    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"]
Nov 02 15:45:03.232746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.232455    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.233197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.233160    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"]
Nov 02 15:45:03.254039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.254009    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qrd\" (UniqueName: \"kubernetes.io/projected/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-kube-api-access-c4qrd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.254817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.254788    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.255305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.255283    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"]
Nov 02 15:45:03.313863 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:03.313823    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98247b5f_4631_4cbe_9fc7_a447913966b7.slice/crio-16842536fe8a9e43fea59e769ef34bdde9f0ca6ae5b7cdb9f5cb7ae7a7896995 WatchSource:0}: Error finding container 16842536fe8a9e43fea59e769ef34bdde9f0ca6ae5b7cdb9f5cb7ae7a7896995: Status 404 returned error can't find the container with id 16842536fe8a9e43fea59e769ef34bdde9f0ca6ae5b7cdb9f5cb7ae7a7896995
Nov 02 15:45:03.357145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.356531    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ed44b3-3268-4076-a279-829f1c7facaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.357145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.356604    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.357145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.356727    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2r77\" (UniqueName: \"kubernetes.io/projected/92ed44b3-3268-4076-a279-829f1c7facaf-kube-api-access-z2r77\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.357145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.356856    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qrd\" (UniqueName: \"kubernetes.io/projected/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-kube-api-access-c4qrd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.357145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.356925    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf8cr\" (UniqueName: \"kubernetes.io/projected/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-kube-api-access-xf8cr\") pod \"maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.357145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.356987    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.357726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.357701    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.408997 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.408958    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"]
Nov 02 15:45:03.426527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.420164    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qrd\" (UniqueName: \"kubernetes.io/projected/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-kube-api-access-c4qrd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.452342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.447794    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"]
Nov 02 15:45:03.460129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.460096    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf8cr\" (UniqueName: \"kubernetes.io/projected/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-kube-api-access-xf8cr\") pod \"maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.460398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.460375    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ed44b3-3268-4076-a279-829f1c7facaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.460477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.460433    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.460567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.460548    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2r77\" (UniqueName: \"kubernetes.io/projected/92ed44b3-3268-4076-a279-829f1c7facaf-kube-api-access-z2r77\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.462585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.461913    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.462585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.462518    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ed44b3-3268-4076-a279-829f1c7facaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.527603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.523801    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:03.527938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.526126    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2r77\" (UniqueName: \"kubernetes.io/projected/92ed44b3-3268-4076-a279-829f1c7facaf-kube-api-access-z2r77\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.528041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.526237    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"]
Nov 02 15:45:03.528041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.527423    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf8cr\" (UniqueName: \"kubernetes.io/projected/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-kube-api-access-xf8cr\") pod \"maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.531748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.531228    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"]
Nov 02 15:45:03.555864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.555832    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:03.593672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.578300    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:03.676185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.676150    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 15:45:03.676447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.676417    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 15:45:03.775850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.775803    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"]
Nov 02 15:45:03.823797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:03.823529    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"]
Nov 02 15:45:04.380008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.379955    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:04.381712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.381687    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"]
Nov 02 15:45:04.381934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.381919    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"]
Nov 02 15:45:04.382059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.382047    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"]
Nov 02 15:45:04.382183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.382172    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"]
Nov 02 15:45:04.382300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.382288    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"]
Nov 02 15:45:04.382396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.382386    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"]
Nov 02 15:45:04.382494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.382485    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1617c52-bvnvp"]
Nov 02 15:45:04.382600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.382590    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a966335-vz2zz"]
Nov 02 15:45:04.455474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.453936    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"]
Nov 02 15:45:04.467301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.466484    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489173    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489256    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489323    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489349    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489415    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bff311-0d0f-423b-afd4-a80671e90fde-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489452    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489509    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489540    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bff311-0d0f-423b-afd4-a80671e90fde-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.489775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.489593    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxlz\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-kube-api-access-5cxlz\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.502833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.466373    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"]
Nov 02 15:45:04.528150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.528076    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"]
Nov 02 15:45:04.528402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.528385    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"]
Nov 02 15:45:04.528527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.528514    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"]
Nov 02 15:45:04.528716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.528701    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"]
Nov 02 15:45:04.528949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.528935    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.551967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.545192    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" event={"ID":"75fa808b-30f1-4090-859a-c8bb8fd9a999","Type":"ContainerStarted","Data":"a56e24a6de7eb9c59ec893a6bb7e741a2c186cf89a976c7389d4ca2fb39ae00e"}
Nov 02 15:45:04.554970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.554490    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" event={"ID":"b568d3ce-7e0d-4909-8835-f2efc6bd4793","Type":"ContainerStarted","Data":"9018dc3b7f9497cddf32512d61bd313d04738d7ab18b6c097851c0272e210d39"}
Nov 02 15:45:04.556775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.556691    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" event={"ID":"08cc86fb-8725-48e1-9a73-7ad63bc05755","Type":"ContainerStarted","Data":"aa16a546dcd84775a6501a7257f6022e412144170e8f08dc034413af5424163f"}
Nov 02 15:45:04.559505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.559115    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerStarted","Data":"42841d0b481840c5516ec7144ffcba4f20353113d882cbc42225a13616899848"}
Nov 02 15:45:04.584115 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:04.579142    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767a5bf4_2b1b_494d_bb0f_d9fca4e295d7.slice/crio-3713ec19eeb6cc3fd7e49c8478d2402b9399681d9189ce761e9835b63246bc4a WatchSource:0}: Error finding container 3713ec19eeb6cc3fd7e49c8478d2402b9399681d9189ce761e9835b63246bc4a: Status 404 returned error can't find the container with id 3713ec19eeb6cc3fd7e49c8478d2402b9399681d9189ce761e9835b63246bc4a
Nov 02 15:45:04.605886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.596367    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.605886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.596412    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.605886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.596490    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bff311-0d0f-423b-afd4-a80671e90fde-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.605886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.596556    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.605886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.596614    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.605886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.602038    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.605886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.602992    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bff311-0d0f-423b-afd4-a80671e90fde-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.608965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.606610    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.608965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.608322    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bff311-0d0f-423b-afd4-a80671e90fde-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.608965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.608396    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxlz\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-kube-api-access-5cxlz\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.608965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.608565    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.608965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.608696    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.609338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.609033    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.613274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.609895    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.613274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.612898    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.614397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.614352    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bff311-0d0f-423b-afd4-a80671e90fde-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.621519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.621491    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" event={"ID":"98247b5f-4631-4cbe-9fc7-a447913966b7","Type":"ContainerStarted","Data":"16842536fe8a9e43fea59e769ef34bdde9f0ca6ae5b7cdb9f5cb7ae7a7896995"}
Nov 02 15:45:04.666504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.666428    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.686772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.686739    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxlz\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-kube-api-access-5cxlz\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.687162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.687040    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body=
Nov 02 15:45:04.687162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.687100    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused"
Nov 02 15:45:04.691892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.691863    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" event={"ID":"e4c120c3-da25-4a39-8a28-916dce067311","Type":"ContainerStarted","Data":"6453625d5cbc976865011d65eff483794eebb9a6df4cb9580e5559bf92896ce6"}
Nov 02 15:45:04.692071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.692054    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" event={"ID":"e4c120c3-da25-4a39-8a28-916dce067311","Type":"ContainerStarted","Data":"6bc4e1db1f12b996de20e9f61bd5e1dbc2b16b9a1aa6487f65f21914746d2359"}
Nov 02 15:45:04.723347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.723298    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.723703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.723625    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.723874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.723855    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc7ae13-d1f3-416b-92bd-4bec39365f43-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.724033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.724016    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.724151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.724137    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.724258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.724241    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.724397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.724380    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.724566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.724551    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rqr\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-kube-api-access-t5rqr\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.724852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.724832    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.790682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.790609    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" podStartSLOduration=4.790585683 podStartE2EDuration="4.790585683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:04.756364018 +0000 UTC m=+2899.010317644" watchObservedRunningTime="2026-11-02 15:45:04.790585683 +0000 UTC m=+2899.044539302"
Nov 02 15:45:04.807622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.807466    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:04.843404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.842631    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.843855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.843430    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-credential-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844174    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844353    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844389    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc7ae13-d1f3-416b-92bd-4bec39365f43-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844439    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844486    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844512    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844561    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.846165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.844659    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rqr\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-kube-api-access-t5rqr\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.848247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.847773    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-certs\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.848247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.847847    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-data\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.848247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.848205    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-socket\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.851795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.848866    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc7ae13-d1f3-416b-92bd-4bec39365f43-istiod-ca-cert\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.861853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.861796    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-envoy\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.862602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.862121    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-podinfo\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.869489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.868908    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"]
Nov 02 15:45:04.878721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.874559    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" podStartSLOduration=4.874542521 podStartE2EDuration="4.874542521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:04.842507164 +0000 UTC m=+2899.096460788" watchObservedRunningTime="2026-11-02 15:45:04.874542521 +0000 UTC m=+2899.128496147"
Nov 02 15:45:04.879425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.879367    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-token\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.879759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.879631    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rqr\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-kube-api-access-t5rqr\") pod \"e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") " pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.917136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:04.917100    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:04.970288 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:04.970239    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ed44b3_3268_4076_a279_829f1c7facaf.slice/crio-e8a629a34bb1bcf0c6fee74aa8f93048def16a244e94324d749e012d870621b5 WatchSource:0}: Error finding container e8a629a34bb1bcf0c6fee74aa8f93048def16a244e94324d749e012d870621b5: Status 404 returned error can't find the container with id e8a629a34bb1bcf0c6fee74aa8f93048def16a244e94324d749e012d870621b5
Nov 02 15:45:05.320797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.320771    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"]
Nov 02 15:45:05.324807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.324768    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:05.324903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.324848    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:05.324903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.324891    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:05.575639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.575353    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"]
Nov 02 15:45:05.586741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.585271    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:05.586741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.585367    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:05.586741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.585406    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:05.680715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.680636    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:45:05.681781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.681753    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:45:05.682914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.682891    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:45:05.708441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.708411    7019 generic.go:356] "Generic (PLEG): container finished" podID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerID="4ad96d7e7b8c4e55a1229772dfdeab1547ef2f701e150c09a1a24ae0f9fab8a5" exitCode=60
Nov 02 15:45:05.708576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.708538    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" event={"ID":"75fa808b-30f1-4090-859a-c8bb8fd9a999","Type":"ContainerDied","Data":"4ad96d7e7b8c4e55a1229772dfdeab1547ef2f701e150c09a1a24ae0f9fab8a5"}
Nov 02 15:45:05.708924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.708905    7019 scope.go:122] "RemoveContainer" containerID="4ad96d7e7b8c4e55a1229772dfdeab1547ef2f701e150c09a1a24ae0f9fab8a5"
Nov 02 15:45:05.715536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.715495    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" event={"ID":"acc7ae13-d1f3-416b-92bd-4bec39365f43","Type":"ContainerStarted","Data":"5ea3b95570c6debbec3e0177282f7a90296beb3559b19bb85b7721d131e90ecd"}
Nov 02 15:45:05.722280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.722255    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerStarted","Data":"dbc7f30ede80877dc057965ab7ff502283fab0922a1a98e0ddf8192e8557ba12"}
Nov 02 15:45:05.722454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.722438    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerStarted","Data":"102d6de36408b031dc75cf458baa99df4b05d032d9f74bd3bde2e07df77c51f5"}
Nov 02 15:45:05.724719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.724696    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerStarted","Data":"e94623f968c72fc209c80ce841f35ac484a15df5cf8b56c0add7e37b38bd9a8e"}
Nov 02 15:45:05.724823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.724742    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerStarted","Data":"1773687e6c84cc0b0b5e21326936d6db3c82f7e31f22d2b4d93be5814147c5b2"}
Nov 02 15:45:05.729718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.727202    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerStarted","Data":"2d1f542957bd39fcf7715833b5771dcec671eb1fb28c8a4d4c5094d88a25247d"}
Nov 02 15:45:05.729718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.727249    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerStarted","Data":"3713ec19eeb6cc3fd7e49c8478d2402b9399681d9189ce761e9835b63246bc4a"}
Nov 02 15:45:05.730712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.730688    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerStarted","Data":"f5e3ee806ea285dcd3e9a503cd58f00ecb7b70478a6453fdfabfcd7f65688e5f"}
Nov 02 15:45:05.730819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.730735    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerStarted","Data":"b6232ad2e01ec60691222f974de329f2410739f156749cdef211463ff96d5909"}
Nov 02 15:45:05.733111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.733089    7019 generic.go:356] "Generic (PLEG): container finished" podID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerID="7c6d74b9a91c29c76937c47e2cc4406edd994a62d913abc1f60adab46cb5ab85" exitCode=60
Nov 02 15:45:05.733306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.733281    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" event={"ID":"b568d3ce-7e0d-4909-8835-f2efc6bd4793","Type":"ContainerDied","Data":"7c6d74b9a91c29c76937c47e2cc4406edd994a62d913abc1f60adab46cb5ab85"}
Nov 02 15:45:05.733685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.733666    7019 scope.go:122] "RemoveContainer" containerID="7c6d74b9a91c29c76937c47e2cc4406edd994a62d913abc1f60adab46cb5ab85"
Nov 02 15:45:05.745298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.745272    7019 generic.go:356] "Generic (PLEG): container finished" podID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerID="c5c85e2c20a259cac26aabbd0d88cbf14bc4b27780e4dd75e45cbffb2c93fbf1" exitCode=60
Nov 02 15:45:05.746281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.745745    7019 scope.go:122] "RemoveContainer" containerID="c5c85e2c20a259cac26aabbd0d88cbf14bc4b27780e4dd75e45cbffb2c93fbf1"
Nov 02 15:45:05.746281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.745762    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" event={"ID":"08cc86fb-8725-48e1-9a73-7ad63bc05755","Type":"ContainerDied","Data":"c5c85e2c20a259cac26aabbd0d88cbf14bc4b27780e4dd75e45cbffb2c93fbf1"}
Nov 02 15:45:05.756211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.755476    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerStarted","Data":"1879050e54051cb1856939676bfe0ae70bae0f7dae72b1ab9ea96ba0b4f080af"}
Nov 02 15:45:05.756211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.755531    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerStarted","Data":"e8a629a34bb1bcf0c6fee74aa8f93048def16a244e94324d749e012d870621b5"}
Nov 02 15:45:05.759062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.759042    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" event={"ID":"24bff311-0d0f-423b-afd4-a80671e90fde","Type":"ContainerStarted","Data":"25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384"}
Nov 02 15:45:05.759191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.759178    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" event={"ID":"24bff311-0d0f-423b-afd4-a80671e90fde","Type":"ContainerStarted","Data":"7639d4a2897c8fac8402c18a288ca909d108ec07ed2a64ec16b29d01836b1924"}
Nov 02 15:45:05.761092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.761073    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerStarted","Data":"a8cb44438df545550c6d9bc1616c82c47b2248d2d4dcdc92a865b85003afee1d"}
Nov 02 15:45:05.761212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.761200    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerStarted","Data":"9f79c877f9e425934ca742fd4e728e81f8c573b20ede9d2915567e793776ab6d"}
Nov 02 15:45:05.781020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.780987    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerStarted","Data":"b724bcc6c3e8b6968b2f49b46bb4fd1c649d15a527818cff2f4176d66d02a547"}
Nov 02 15:45:05.790056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.788739    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" podStartSLOduration=5.788720862 podStartE2EDuration="5.788720862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:05.78413274 +0000 UTC m=+2900.038086368" watchObservedRunningTime="2026-11-02 15:45:05.788720862 +0000 UTC m=+2900.042674480"
Nov 02 15:45:05.793771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.793739    7019 generic.go:356] "Generic (PLEG): container finished" podID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerID="c57f6a43bb1750f975963e16d0c9a604e5a4a136e72a594da11488892ce7eb39" exitCode=60
Nov 02 15:45:05.794002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.793981    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" event={"ID":"98247b5f-4631-4cbe-9fc7-a447913966b7","Type":"ContainerDied","Data":"c57f6a43bb1750f975963e16d0c9a604e5a4a136e72a594da11488892ce7eb39"}
Nov 02 15:45:05.794632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.794613    7019 scope.go:122] "RemoveContainer" containerID="c57f6a43bb1750f975963e16d0c9a604e5a4a136e72a594da11488892ce7eb39"
Nov 02 15:45:05.802894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.802784    7019 generic.go:356] "Generic (PLEG): container finished" podID="e4c120c3-da25-4a39-8a28-916dce067311" containerID="6453625d5cbc976865011d65eff483794eebb9a6df4cb9580e5559bf92896ce6" exitCode=60
Nov 02 15:45:05.802894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.802867    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" event={"ID":"e4c120c3-da25-4a39-8a28-916dce067311","Type":"ContainerDied","Data":"6453625d5cbc976865011d65eff483794eebb9a6df4cb9580e5559bf92896ce6"}
Nov 02 15:45:05.803194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.803173    7019 scope.go:122] "RemoveContainer" containerID="6453625d5cbc976865011d65eff483794eebb9a6df4cb9580e5559bf92896ce6"
Nov 02 15:45:05.807662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.807611    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerStarted","Data":"e9eea811c29a01a6dc96ebd0d741182c39ec7fcd62a9edc239933e32d8136874"}
Nov 02 15:45:05.817502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.817442    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerStarted","Data":"a9c7974813522f9f92bc205e4369e1c2bc3e118be62be6be4523313edf5524fc"}
Nov 02 15:45:05.817705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.817688    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:05.885329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.881876    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" podStartSLOduration=5.881841112 podStartE2EDuration="5.881841112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:05.837664029 +0000 UTC m=+2900.091617650" watchObservedRunningTime="2026-11-02 15:45:05.881841112 +0000 UTC m=+2900.135794737"
Nov 02 15:45:05.918628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.918586    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a54855-6a17-40d4-bc1a-b01ac5377922" path="/var/lib/kubelet/pods/29a54855-6a17-40d4-bc1a-b01ac5377922/volumes"
Nov 02 15:45:05.923813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.919576    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d788da-4459-4c88-a032-abebe55690f5" path="/var/lib/kubelet/pods/43d788da-4459-4c88-a032-abebe55690f5/volumes"
Nov 02 15:45:05.963884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:05.962414    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" podStartSLOduration=5.96239753 podStartE2EDuration="5.96239753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:05.961134626 +0000 UTC m=+2900.215088274" watchObservedRunningTime="2026-11-02 15:45:05.96239753 +0000 UTC m=+2900.216351154"
Nov 02 15:45:06.060076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.059927    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:06.110309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.101547    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" podStartSLOduration=6.101524545 podStartE2EDuration="6.101524545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.09679599 +0000 UTC m=+2900.350749639" watchObservedRunningTime="2026-11-02 15:45:06.101524545 +0000 UTC m=+2900.355478171"
Nov 02 15:45:06.239634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.239570    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" podStartSLOduration=6.239538817 podStartE2EDuration="6.239538817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.237940657 +0000 UTC m=+2900.491894282" watchObservedRunningTime="2026-11-02 15:45:06.239538817 +0000 UTC m=+2900.493492442"
Nov 02 15:45:06.418133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.417507    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" podStartSLOduration=6.417482855 podStartE2EDuration="6.417482855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.406635719 +0000 UTC m=+2900.660589344" watchObservedRunningTime="2026-11-02 15:45:06.417482855 +0000 UTC m=+2900.671436480"
Nov 02 15:45:06.639708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.632928    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" podStartSLOduration=6.632906891 podStartE2EDuration="6.632906891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.534966883 +0000 UTC m=+2900.788920507" watchObservedRunningTime="2026-11-02 15:45:06.632906891 +0000 UTC m=+2900.886860516"
Nov 02 15:45:06.642084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.642014    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" podStartSLOduration=6.641995017 podStartE2EDuration="6.641995017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.614592463 +0000 UTC m=+2900.868546079" watchObservedRunningTime="2026-11-02 15:45:06.641995017 +0000 UTC m=+2900.895948642"
Nov 02 15:45:06.722703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.714828    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podStartSLOduration=2.714805852 podStartE2EDuration="2.714805852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.70850526 +0000 UTC m=+2900.962458885" watchObservedRunningTime="2026-11-02 15:45:06.714805852 +0000 UTC m=+2900.968759477"
Nov 02 15:45:06.821672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.816245    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 15:45:06.821672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.816324    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 15:45:06.887878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.887838    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" event={"ID":"b568d3ce-7e0d-4909-8835-f2efc6bd4793","Type":"ContainerStarted","Data":"cf061d60252cd15291e3d3c9d8f0595ba4d50c14d7b9edf647562c37ead02ccd"}
Nov 02 15:45:06.909954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.909860    7019 generic.go:356] "Generic (PLEG): container finished" podID="92ed44b3-3268-4076-a279-829f1c7facaf" containerID="1879050e54051cb1856939676bfe0ae70bae0f7dae72b1ab9ea96ba0b4f080af" exitCode=60
Nov 02 15:45:06.910256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.910230    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerDied","Data":"1879050e54051cb1856939676bfe0ae70bae0f7dae72b1ab9ea96ba0b4f080af"}
Nov 02 15:45:06.910919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.910899    7019 scope.go:122] "RemoveContainer" containerID="1879050e54051cb1856939676bfe0ae70bae0f7dae72b1ab9ea96ba0b4f080af"
Nov 02 15:45:06.925488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.925455    7019 generic.go:356] "Generic (PLEG): container finished" podID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerID="a8cb44438df545550c6d9bc1616c82c47b2248d2d4dcdc92a865b85003afee1d" exitCode=60
Nov 02 15:45:06.925767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.925744    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerDied","Data":"a8cb44438df545550c6d9bc1616c82c47b2248d2d4dcdc92a865b85003afee1d"}
Nov 02 15:45:06.926339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.926319    7019 scope.go:122] "RemoveContainer" containerID="a8cb44438df545550c6d9bc1616c82c47b2248d2d4dcdc92a865b85003afee1d"
Nov 02 15:45:06.938616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.938559    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" podStartSLOduration=6.938541692 podStartE2EDuration="6.938541692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:06.938078749 +0000 UTC m=+2901.192032373" watchObservedRunningTime="2026-11-02 15:45:06.938541692 +0000 UTC m=+2901.192495317"
Nov 02 15:45:06.938953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.938926    7019 generic.go:356] "Generic (PLEG): container finished" podID="fa7b87a4-682a-4462-9610-63ee828259e3" containerID="b724bcc6c3e8b6968b2f49b46bb4fd1c649d15a527818cff2f4176d66d02a547" exitCode=60
Nov 02 15:45:06.940901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.939767    7019 scope.go:122] "RemoveContainer" containerID="b724bcc6c3e8b6968b2f49b46bb4fd1c649d15a527818cff2f4176d66d02a547"
Nov 02 15:45:06.940901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.940623    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerDied","Data":"b724bcc6c3e8b6968b2f49b46bb4fd1c649d15a527818cff2f4176d66d02a547"}
Nov 02 15:45:06.950177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.948701    7019 generic.go:356] "Generic (PLEG): container finished" podID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerID="e9eea811c29a01a6dc96ebd0d741182c39ec7fcd62a9edc239933e32d8136874" exitCode=60
Nov 02 15:45:06.950177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.948807    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerDied","Data":"e9eea811c29a01a6dc96ebd0d741182c39ec7fcd62a9edc239933e32d8136874"}
Nov 02 15:45:06.950177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.949186    7019 scope.go:122] "RemoveContainer" containerID="e9eea811c29a01a6dc96ebd0d741182c39ec7fcd62a9edc239933e32d8136874"
Nov 02 15:45:06.960016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.959994    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" event={"ID":"75fa808b-30f1-4090-859a-c8bb8fd9a999","Type":"ContainerStarted","Data":"115c5dda4059462efc4933808d51ff289ec28bab1438c24c83a1721937deb534"}
Nov 02 15:45:06.969054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:06.963824    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" event={"ID":"acc7ae13-d1f3-416b-92bd-4bec39365f43","Type":"ContainerStarted","Data":"448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4"}
Nov 02 15:45:07.020693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.002003    7019 generic.go:356] "Generic (PLEG): container finished" podID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerID="dbc7f30ede80877dc057965ab7ff502283fab0922a1a98e0ddf8192e8557ba12" exitCode=60
Nov 02 15:45:07.020693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.002146    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerDied","Data":"dbc7f30ede80877dc057965ab7ff502283fab0922a1a98e0ddf8192e8557ba12"}
Nov 02 15:45:07.020693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.002689    7019 scope.go:122] "RemoveContainer" containerID="dbc7f30ede80877dc057965ab7ff502283fab0922a1a98e0ddf8192e8557ba12"
Nov 02 15:45:07.049436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.049170    7019 generic.go:356] "Generic (PLEG): container finished" podID="935acbcf-5b16-4096-8a48-73579d2b6505" containerID="e94623f968c72fc209c80ce841f35ac484a15df5cf8b56c0add7e37b38bd9a8e" exitCode=60
Nov 02 15:45:07.049436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.049260    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerDied","Data":"e94623f968c72fc209c80ce841f35ac484a15df5cf8b56c0add7e37b38bd9a8e"}
Nov 02 15:45:07.050135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.049900    7019 scope.go:122] "RemoveContainer" containerID="e94623f968c72fc209c80ce841f35ac484a15df5cf8b56c0add7e37b38bd9a8e"
Nov 02 15:45:07.060137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.060112    7019 generic.go:356] "Generic (PLEG): container finished" podID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerID="2d1f542957bd39fcf7715833b5771dcec671eb1fb28c8a4d4c5094d88a25247d" exitCode=60
Nov 02 15:45:07.060372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.060358    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerDied","Data":"2d1f542957bd39fcf7715833b5771dcec671eb1fb28c8a4d4c5094d88a25247d"}
Nov 02 15:45:07.060511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.060390    7019 scope.go:122] "RemoveContainer" containerID="2d1f542957bd39fcf7715833b5771dcec671eb1fb28c8a4d4c5094d88a25247d"
Nov 02 15:45:07.064166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.064117    7019 generic.go:356] "Generic (PLEG): container finished" podID="73bca953-9136-42bd-916d-1e3f1915b53a" containerID="f5e3ee806ea285dcd3e9a503cd58f00ecb7b70478a6453fdfabfcd7f65688e5f" exitCode=60
Nov 02 15:45:07.085358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.065140    7019 scope.go:122] "RemoveContainer" containerID="f5e3ee806ea285dcd3e9a503cd58f00ecb7b70478a6453fdfabfcd7f65688e5f"
Nov 02 15:45:07.085358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.065487    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerDied","Data":"f5e3ee806ea285dcd3e9a503cd58f00ecb7b70478a6453fdfabfcd7f65688e5f"}
Nov 02 15:45:07.436969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.436916    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podStartSLOduration=3.436901361 podStartE2EDuration="3.436901361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.315306261 +0000 UTC m=+2901.569259886" watchObservedRunningTime="2026-11-02 15:45:07.436901361 +0000 UTC m=+2901.690854984"
Nov 02 15:45:07.642727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.634928    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" podStartSLOduration=7.634906527 podStartE2EDuration="7.634906527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:07.606941822 +0000 UTC m=+2901.860895446" watchObservedRunningTime="2026-11-02 15:45:07.634906527 +0000 UTC m=+2901.888860153"
Nov 02 15:45:07.821973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.821852    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 15:45:07.821973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.821906    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 15:45:07.930724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.919932    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:07.934174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.934062    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 02 15:45:07.934174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:07.934115    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 02 15:45:08.077787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.049539    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:08.257880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.257843    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerStarted","Data":"b484663d301f6e1274d3afe2436884691da8483a9756d88bb15ef9c60336c959"}
Nov 02 15:45:08.261027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.260949    7019 generic.go:356] "Generic (PLEG): container finished" podID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerID="6fe7ce6949aa9b13b3d08e05430ac73cfb658292869e478e82bbaa3002f83d14" exitCode=60
Nov 02 15:45:08.261027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.261012    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" event={"ID":"98247b5f-4631-4cbe-9fc7-a447913966b7","Type":"ContainerDied","Data":"6fe7ce6949aa9b13b3d08e05430ac73cfb658292869e478e82bbaa3002f83d14"}
Nov 02 15:45:08.261195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.261067    7019 scope.go:122] "RemoveContainer" containerID="c57f6a43bb1750f975963e16d0c9a604e5a4a136e72a594da11488892ce7eb39"
Nov 02 15:45:08.261421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.261255    7019 scope.go:122] "RemoveContainer" containerID="6fe7ce6949aa9b13b3d08e05430ac73cfb658292869e478e82bbaa3002f83d14"
Nov 02 15:45:08.261605 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:08.261581    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r_odh-ai-gateway-infra(98247b5f-4631-4cbe-9fc7-a447913966b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7"
Nov 02 15:45:08.263273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.263253    7019 generic.go:356] "Generic (PLEG): container finished" podID="e4c120c3-da25-4a39-8a28-916dce067311" containerID="695812f8650a4fd0a9c96f022b3eb92cddc9d792d2d771a6561975408c3cf201" exitCode=60
Nov 02 15:45:08.263355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.263335    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" event={"ID":"e4c120c3-da25-4a39-8a28-916dce067311","Type":"ContainerDied","Data":"695812f8650a4fd0a9c96f022b3eb92cddc9d792d2d771a6561975408c3cf201"}
Nov 02 15:45:08.263596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.263577    7019 scope.go:122] "RemoveContainer" containerID="695812f8650a4fd0a9c96f022b3eb92cddc9d792d2d771a6561975408c3cf201"
Nov 02 15:45:08.264013 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:08.263985    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5_odh-ai-gateway-infra(e4c120c3-da25-4a39-8a28-916dce067311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" podUID="e4c120c3-da25-4a39-8a28-916dce067311"
Nov 02 15:45:08.305740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.305301    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerStarted","Data":"8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818"}
Nov 02 15:45:08.341742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.341669    7019 scope.go:122] "RemoveContainer" containerID="6453625d5cbc976865011d65eff483794eebb9a6df4cb9580e5559bf92896ce6"
Nov 02 15:45:08.344038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.344009    7019 generic.go:356] "Generic (PLEG): container finished" podID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerID="115c5dda4059462efc4933808d51ff289ec28bab1438c24c83a1721937deb534" exitCode=60
Nov 02 15:45:08.344451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.344155    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" event={"ID":"75fa808b-30f1-4090-859a-c8bb8fd9a999","Type":"ContainerDied","Data":"115c5dda4059462efc4933808d51ff289ec28bab1438c24c83a1721937deb534"}
Nov 02 15:45:08.346159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.346135    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerStarted","Data":"1bb6f670d4ac7f938cae9c9b64be9f461b4b67618f7aca3ac8945fe4e0b6ed35"}
Nov 02 15:45:08.347838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.347811    7019 scope.go:122] "RemoveContainer" containerID="115c5dda4059462efc4933808d51ff289ec28bab1438c24c83a1721937deb534"
Nov 02 15:45:08.348206 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:08.348168    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55_odh-ai-gateway-infra(75fa808b-30f1-4090-859a-c8bb8fd9a999)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999"
Nov 02 15:45:08.349289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.349268    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerStarted","Data":"1f419190702c3b641a82c8335a93c69a495c177b36739ccc5b62b36cbaf4f6f9"}
Nov 02 15:45:08.351705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.351683    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerStarted","Data":"60989db43c764905b0d9799c95c08412a1178d6065e3c5c50d7fcb6c99f47c69"}
Nov 02 15:45:08.355439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.355418    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerStarted","Data":"21a651b3f1b010856796204dc093b56ec0b16eca5bff5be394c7de89a0dbedc6"}
Nov 02 15:45:08.358704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.358627    7019 generic.go:356] "Generic (PLEG): container finished" podID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerID="cf061d60252cd15291e3d3c9d8f0595ba4d50c14d7b9edf647562c37ead02ccd" exitCode=60
Nov 02 15:45:08.358704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.358673    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" event={"ID":"b568d3ce-7e0d-4909-8835-f2efc6bd4793","Type":"ContainerDied","Data":"cf061d60252cd15291e3d3c9d8f0595ba4d50c14d7b9edf647562c37ead02ccd"}
Nov 02 15:45:08.359048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.359029    7019 scope.go:122] "RemoveContainer" containerID="cf061d60252cd15291e3d3c9d8f0595ba4d50c14d7b9edf647562c37ead02ccd"
Nov 02 15:45:08.359354 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:08.359326    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt_odh-ai-gateway-infra(b568d3ce-7e0d-4909-8835-f2efc6bd4793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793"
Nov 02 15:45:08.363840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.363823    7019 scope.go:122] "RemoveContainer" containerID="4ad96d7e7b8c4e55a1229772dfdeab1547ef2f701e150c09a1a24ae0f9fab8a5"
Nov 02 15:45:08.446945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.446784    7019 generic.go:356] "Generic (PLEG): container finished" podID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerID="0d9d08aa178b46e07fdd7b1c43b4e352c3a4404786d8c94ff83f12a71bd87863" exitCode=60
Nov 02 15:45:08.446945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.446879    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" event={"ID":"08cc86fb-8725-48e1-9a73-7ad63bc05755","Type":"ContainerDied","Data":"0d9d08aa178b46e07fdd7b1c43b4e352c3a4404786d8c94ff83f12a71bd87863"}
Nov 02 15:45:08.452840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.452807    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerStarted","Data":"f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23"}
Nov 02 15:45:08.454964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.454062    7019 scope.go:122] "RemoveContainer" containerID="0d9d08aa178b46e07fdd7b1c43b4e352c3a4404786d8c94ff83f12a71bd87863"
Nov 02 15:45:08.454964 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:08.454386    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds_odh-ai-gateway-infra(08cc86fb-8725-48e1-9a73-7ad63bc05755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755"
Nov 02 15:45:08.517675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.494710    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerStarted","Data":"d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37"}
Nov 02 15:45:08.555399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.544177    7019 scope.go:122] "RemoveContainer" containerID="7c6d74b9a91c29c76937c47e2cc4406edd994a62d913abc1f60adab46cb5ab85"
Nov 02 15:45:08.555399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.552269    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:08.564641 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:08.564609    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ed44b3_3268_4076_a279_829f1c7facaf.slice/crio-f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2915eaf5_dbe8_4df0_b6c9_ab65dc888e8b.slice/crio-8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d681fe3_a40e_411a_9a80_8b2c7b90a9d9.slice/crio-conmon-d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:45:08.835889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.835753    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 15:45:08.835889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.835810    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 15:45:08.948670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.945887    7019 scope.go:122] "RemoveContainer" containerID="c5c85e2c20a259cac26aabbd0d88cbf14bc4b27780e4dd75e45cbffb2c93fbf1"
Nov 02 15:45:08.948670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.946046    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 02 15:45:08.948670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:08.946090    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 02 15:45:09.535566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.535529    7019 generic.go:356] "Generic (PLEG): container finished" podID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerID="8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818" exitCode=60
Nov 02 15:45:09.535755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.535625    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerDied","Data":"8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818"}
Nov 02 15:45:09.535755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.535682    7019 scope.go:122] "RemoveContainer" containerID="e9eea811c29a01a6dc96ebd0d741182c39ec7fcd62a9edc239933e32d8136874"
Nov 02 15:45:09.536081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.536065    7019 scope.go:122] "RemoveContainer" containerID="8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818"
Nov 02 15:45:09.536430 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.536413    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t_odh-ai-gateway-infra(2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b"
Nov 02 15:45:09.538682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.538660    7019 scope.go:122] "RemoveContainer" containerID="115c5dda4059462efc4933808d51ff289ec28bab1438c24c83a1721937deb534"
Nov 02 15:45:09.539135 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.539112    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55_odh-ai-gateway-infra(75fa808b-30f1-4090-859a-c8bb8fd9a999)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999"
Nov 02 15:45:09.541458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.541433    7019 generic.go:356] "Generic (PLEG): container finished" podID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerID="1bb6f670d4ac7f938cae9c9b64be9f461b4b67618f7aca3ac8945fe4e0b6ed35" exitCode=60
Nov 02 15:45:09.541554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.541513    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerDied","Data":"1bb6f670d4ac7f938cae9c9b64be9f461b4b67618f7aca3ac8945fe4e0b6ed35"}
Nov 02 15:45:09.541870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.541850    7019 scope.go:122] "RemoveContainer" containerID="1bb6f670d4ac7f938cae9c9b64be9f461b4b67618f7aca3ac8945fe4e0b6ed35"
Nov 02 15:45:09.542282 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.542255    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt_odh-ai-gateway-infra(1a625702-a861-4efb-9bf5-b50c4f3119cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf"
Nov 02 15:45:09.544499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.544471    7019 generic.go:356] "Generic (PLEG): container finished" podID="935acbcf-5b16-4096-8a48-73579d2b6505" containerID="1f419190702c3b641a82c8335a93c69a495c177b36739ccc5b62b36cbaf4f6f9" exitCode=60
Nov 02 15:45:09.544597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.544556    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerDied","Data":"1f419190702c3b641a82c8335a93c69a495c177b36739ccc5b62b36cbaf4f6f9"}
Nov 02 15:45:09.545023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.545006    7019 scope.go:122] "RemoveContainer" containerID="1f419190702c3b641a82c8335a93c69a495c177b36739ccc5b62b36cbaf4f6f9"
Nov 02 15:45:09.545273 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.545253    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z_odh-ai-gateway-infra(935acbcf-5b16-4096-8a48-73579d2b6505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" podUID="935acbcf-5b16-4096-8a48-73579d2b6505"
Nov 02 15:45:09.547446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.547427    7019 generic.go:356] "Generic (PLEG): container finished" podID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerID="60989db43c764905b0d9799c95c08412a1178d6065e3c5c50d7fcb6c99f47c69" exitCode=60
Nov 02 15:45:09.547520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.547501    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerDied","Data":"60989db43c764905b0d9799c95c08412a1178d6065e3c5c50d7fcb6c99f47c69"}
Nov 02 15:45:09.547741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.547723    7019 scope.go:122] "RemoveContainer" containerID="60989db43c764905b0d9799c95c08412a1178d6065e3c5c50d7fcb6c99f47c69"
Nov 02 15:45:09.547952 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.547935    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf_odh-ai-gateway-infra(767a5bf4-2b1b-494d-bb0f-d9fca4e295d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7"
Nov 02 15:45:09.549389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.549370    7019 generic.go:356] "Generic (PLEG): container finished" podID="73bca953-9136-42bd-916d-1e3f1915b53a" containerID="21a651b3f1b010856796204dc093b56ec0b16eca5bff5be394c7de89a0dbedc6" exitCode=60
Nov 02 15:45:09.549494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.549446    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerDied","Data":"21a651b3f1b010856796204dc093b56ec0b16eca5bff5be394c7de89a0dbedc6"}
Nov 02 15:45:09.549727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.549709    7019 scope.go:122] "RemoveContainer" containerID="21a651b3f1b010856796204dc093b56ec0b16eca5bff5be394c7de89a0dbedc6"
Nov 02 15:45:09.550011 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.549989    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp_odh-ai-gateway-infra(73bca953-9136-42bd-916d-1e3f1915b53a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" podUID="73bca953-9136-42bd-916d-1e3f1915b53a"
Nov 02 15:45:09.551280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.551263    7019 scope.go:122] "RemoveContainer" containerID="cf061d60252cd15291e3d3c9d8f0595ba4d50c14d7b9edf647562c37ead02ccd"
Nov 02 15:45:09.551550 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.551524    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt_odh-ai-gateway-infra(b568d3ce-7e0d-4909-8835-f2efc6bd4793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793"
Nov 02 15:45:09.552933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.552918    7019 scope.go:122] "RemoveContainer" containerID="0d9d08aa178b46e07fdd7b1c43b4e352c3a4404786d8c94ff83f12a71bd87863"
Nov 02 15:45:09.553144 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.553120    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds_odh-ai-gateway-infra(08cc86fb-8725-48e1-9a73-7ad63bc05755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755"
Nov 02 15:45:09.553780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.553760    7019 scope.go:122] "RemoveContainer" containerID="dbc7f30ede80877dc057965ab7ff502283fab0922a1a98e0ddf8192e8557ba12"
Nov 02 15:45:09.554882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.554864    7019 generic.go:356] "Generic (PLEG): container finished" podID="92ed44b3-3268-4076-a279-829f1c7facaf" containerID="f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23" exitCode=60
Nov 02 15:45:09.555075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.555053    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerDied","Data":"f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23"}
Nov 02 15:45:09.555160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.555145    7019 scope.go:122] "RemoveContainer" containerID="f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23"
Nov 02 15:45:09.555354 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.555336    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6_odh-ai-gateway-infra(92ed44b3-3268-4076-a279-829f1c7facaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" podUID="92ed44b3-3268-4076-a279-829f1c7facaf"
Nov 02 15:45:09.557256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.557211    7019 generic.go:356] "Generic (PLEG): container finished" podID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerID="d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37" exitCode=60
Nov 02 15:45:09.557352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.557292    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerDied","Data":"d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37"}
Nov 02 15:45:09.557493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.557477    7019 scope.go:122] "RemoveContainer" containerID="d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37"
Nov 02 15:45:09.557723 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.557705    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s_odh-ai-gateway-infra(3d681fe3-a40e-411a-9a80-8b2c7b90a9d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9"
Nov 02 15:45:09.559187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.559167    7019 generic.go:356] "Generic (PLEG): container finished" podID="fa7b87a4-682a-4462-9610-63ee828259e3" containerID="b484663d301f6e1274d3afe2436884691da8483a9756d88bb15ef9c60336c959" exitCode=60
Nov 02 15:45:09.559309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.559262    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerDied","Data":"b484663d301f6e1274d3afe2436884691da8483a9756d88bb15ef9c60336c959"}
Nov 02 15:45:09.559477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.559458    7019 scope.go:122] "RemoveContainer" containerID="b484663d301f6e1274d3afe2436884691da8483a9756d88bb15ef9c60336c959"
Nov 02 15:45:09.559724 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.559703    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z_odh-ai-gateway-infra(fa7b87a4-682a-4462-9610-63ee828259e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" podUID="fa7b87a4-682a-4462-9610-63ee828259e3"
Nov 02 15:45:09.561041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.561024    7019 scope.go:122] "RemoveContainer" containerID="6fe7ce6949aa9b13b3d08e05430ac73cfb658292869e478e82bbaa3002f83d14"
Nov 02 15:45:09.561297 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.561270    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r_odh-ai-gateway-infra(98247b5f-4631-4cbe-9fc7-a447913966b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7"
Nov 02 15:45:09.562737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.562718    7019 scope.go:122] "RemoveContainer" containerID="695812f8650a4fd0a9c96f022b3eb92cddc9d792d2d771a6561975408c3cf201"
Nov 02 15:45:09.562992 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:09.562973    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5_odh-ai-gateway-infra(e4c120c3-da25-4a39-8a28-916dce067311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" podUID="e4c120c3-da25-4a39-8a28-916dce067311"
Nov 02 15:45:09.627035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.618879    7019 scope.go:122] "RemoveContainer" containerID="e94623f968c72fc209c80ce841f35ac484a15df5cf8b56c0add7e37b38bd9a8e"
Nov 02 15:45:09.658553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.658490    7019 scope.go:122] "RemoveContainer" containerID="2d1f542957bd39fcf7715833b5771dcec671eb1fb28c8a4d4c5094d88a25247d"
Nov 02 15:45:09.738830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.738083    7019 scope.go:122] "RemoveContainer" containerID="f5e3ee806ea285dcd3e9a503cd58f00ecb7b70478a6453fdfabfcd7f65688e5f"
Nov 02 15:45:09.755619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.755591    7019 scope.go:122] "RemoveContainer" containerID="1879050e54051cb1856939676bfe0ae70bae0f7dae72b1ab9ea96ba0b4f080af"
Nov 02 15:45:09.841739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.836125    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 15:45:09.841739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.836190    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 15:45:09.861261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.861233    7019 scope.go:122] "RemoveContainer" containerID="a8cb44438df545550c6d9bc1616c82c47b2248d2d4dcdc92a865b85003afee1d"
Nov 02 15:45:09.920017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.919982    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 02 15:45:09.920423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.920391    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 02 15:45:09.976026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:09.975994    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:10.054200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.054177    7019 scope.go:122] "RemoveContainer" containerID="b724bcc6c3e8b6968b2f49b46bb4fd1c649d15a527818cff2f4176d66d02a547"
Nov 02 15:45:10.651785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.651756    7019 scope.go:122] "RemoveContainer" containerID="21a651b3f1b010856796204dc093b56ec0b16eca5bff5be394c7de89a0dbedc6"
Nov 02 15:45:10.652423 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.652390    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp_odh-ai-gateway-infra(73bca953-9136-42bd-916d-1e3f1915b53a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" podUID="73bca953-9136-42bd-916d-1e3f1915b53a"
Nov 02 15:45:10.654786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.654711    7019 scope.go:122] "RemoveContainer" containerID="f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23"
Nov 02 15:45:10.655148 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.655105    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6_odh-ai-gateway-infra(92ed44b3-3268-4076-a279-829f1c7facaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" podUID="92ed44b3-3268-4076-a279-829f1c7facaf"
Nov 02 15:45:10.656894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.656840    7019 scope.go:122] "RemoveContainer" containerID="d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37"
Nov 02 15:45:10.657117 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.657093    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s_odh-ai-gateway-infra(3d681fe3-a40e-411a-9a80-8b2c7b90a9d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9"
Nov 02 15:45:10.659641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.659478    7019 scope.go:122] "RemoveContainer" containerID="b484663d301f6e1274d3afe2436884691da8483a9756d88bb15ef9c60336c959"
Nov 02 15:45:10.660017 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.659979    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z_odh-ai-gateway-infra(fa7b87a4-682a-4462-9610-63ee828259e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" podUID="fa7b87a4-682a-4462-9610-63ee828259e3"
Nov 02 15:45:10.660944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.660924    7019 scope.go:122] "RemoveContainer" containerID="8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818"
Nov 02 15:45:10.661132 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.661113    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t_odh-ai-gateway-infra(2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b"
Nov 02 15:45:10.662323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.662304    7019 scope.go:122] "RemoveContainer" containerID="1bb6f670d4ac7f938cae9c9b64be9f461b4b67618f7aca3ac8945fe4e0b6ed35"
Nov 02 15:45:10.662535 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.662517    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt_odh-ai-gateway-infra(1a625702-a861-4efb-9bf5-b50c4f3119cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf"
Nov 02 15:45:10.663843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.663825    7019 scope.go:122] "RemoveContainer" containerID="1f419190702c3b641a82c8335a93c69a495c177b36739ccc5b62b36cbaf4f6f9"
Nov 02 15:45:10.664059 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.664036    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z_odh-ai-gateway-infra(935acbcf-5b16-4096-8a48-73579d2b6505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" podUID="935acbcf-5b16-4096-8a48-73579d2b6505"
Nov 02 15:45:10.729736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.728008    7019 scope.go:122] "RemoveContainer" containerID="60989db43c764905b0d9799c95c08412a1178d6065e3c5c50d7fcb6c99f47c69"
Nov 02 15:45:10.729736 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:10.728458    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf_odh-ai-gateway-infra(767a5bf4-2b1b-494d-bb0f-d9fca4e295d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7"
Nov 02 15:45:10.821672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.818243    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 15:45:10.821672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.818302    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 15:45:10.932731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.928697    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 02 15:45:10.932731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:10.928786    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 02 15:45:11.407936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.407894    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"]
Nov 02 15:45:11.450182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.450145    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"]
Nov 02 15:45:11.450329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.450289    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.551519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.551481    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cb0ace-10e5-4702-8d93-029c41e82e1f-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.551698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.551548    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqqn\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-kube-api-access-7qqqn\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.551820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.551792    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.551918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.551901    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.552002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.551988    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.552037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.552017    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.552070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.552060    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.552139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.552091    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.552139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.552110    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.653957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.653922    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqqn\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-kube-api-access-7qqqn\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654045    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654125    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654209    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654232    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654269    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654303    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654321    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654408    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cb0ace-10e5-4702-8d93-029c41e82e1f-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654633    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-data\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.654755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.654701    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-credential-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.655101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.655078    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-socket\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.655198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.655151    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cb0ace-10e5-4702-8d93-029c41e82e1f-istiod-ca-cert\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.655256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.655216    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-certs\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.657390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.657365    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-podinfo\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.657483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.657407    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-envoy\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.719312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.718868    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqqn\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-kube-api-access-7qqqn\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.721847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.721819    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-token\") pod \"e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") " pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.748681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.748629    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:11.799672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.795360    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:11.816756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.816704    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body=
Nov 02 15:45:11.816968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.816776    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused"
Nov 02 15:45:11.921622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.921573    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body=
Nov 02 15:45:11.921899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:11.921628    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused"
Nov 02 15:45:12.394670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.370312    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:12.455609 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:12.454574    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cb0ace_10e5_4702_8d93_029c41e82e1f.slice/crio-12db2a994b1527d90e0bc800e6de5bec44bc181334a43960b983425fdf8a2724 WatchSource:0}: Error finding container 12db2a994b1527d90e0bc800e6de5bec44bc181334a43960b983425fdf8a2724: Status 404 returned error can't find the container with id 12db2a994b1527d90e0bc800e6de5bec44bc181334a43960b983425fdf8a2724
Nov 02 15:45:12.460243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.459178    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:12.460243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.459281    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:12.460243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.459327    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:12.536834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.528940    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"]
Nov 02 15:45:12.608718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.590406    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:12.625854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.625701    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"]
Nov 02 15:45:12.635742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.635118    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.635957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.635804    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"]
Nov 02 15:45:12.665008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.664984    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a55a55-openshift-default-dockercfg-bkd2n\""
Nov 02 15:45:12.710278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710247    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-credential-socket\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.710432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710340    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-certs\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.710518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710502    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-envoy\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.710578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710548    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02302e75-aacc-4653-b806-4f06991d24c3-istio-podinfo\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.710774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710757    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-socket\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.710872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710803    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-data\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.710973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710956    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm26c\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-kube-api-access-xm26c\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.711035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.710992    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02302e75-aacc-4653-b806-4f06991d24c3-istiod-ca-cert\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.711088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.711029    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-istio-token\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.750102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.750071    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" event={"ID":"68cb0ace-10e5-4702-8d93-029c41e82e1f","Type":"ContainerStarted","Data":"2e149037318bcef30d0bf49a8029303425e881ed8e85e2cf39cae706ea532717"}
Nov 02 15:45:12.750343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.750322    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" event={"ID":"68cb0ace-10e5-4702-8d93-029c41e82e1f","Type":"ContainerStarted","Data":"12db2a994b1527d90e0bc800e6de5bec44bc181334a43960b983425fdf8a2724"}
Nov 02 15:45:12.804723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.803693    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podStartSLOduration=1.8036742019999998 podStartE2EDuration="1.803674202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:12.802020847 +0000 UTC m=+2907.055974472" watchObservedRunningTime="2026-11-02 15:45:12.803674202 +0000 UTC m=+2907.057627830"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818205    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-istio-token\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818284    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-credential-socket\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818339    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-certs\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818426    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-envoy\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818453    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02302e75-aacc-4653-b806-4f06991d24c3-istio-podinfo\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818583    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-socket\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818615    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-data\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818722    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm26c\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-kube-api-access-xm26c\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.818741    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02302e75-aacc-4653-b806-4f06991d24c3-istiod-ca-cert\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.819311    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02302e75-aacc-4653-b806-4f06991d24c3-istiod-ca-cert\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.819758    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-credential-socket\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.820086    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-certs\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.820626    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-socket\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.821613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.821511    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-data\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.822603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.821892    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:12.823002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.822984    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:12.823193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.823176    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:12.824104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.824087    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:45:12.825149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.825125    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-envoy\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.828482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.828440    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02302e75-aacc-4653-b806-4f06991d24c3-istio-podinfo\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.847179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.847152    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-istio-token\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.850017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.849759    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm26c\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-kube-api-access-xm26c\") pod \"e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") " pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:12.926694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.925314    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:12.926694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.926322    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:12.927403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.927383    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:45:12.996671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:12.996158    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:13.367229 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.367196    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"]
Nov 02 15:45:13.371077 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:13.371046    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02302e75_aacc_4653_b806_4f06991d24c3.slice/crio-191610b24a51bf60e1910e1c30f4e4c214ec52ce76c59d8d06d56b52cbe2c10a WatchSource:0}: Error finding container 191610b24a51bf60e1910e1c30f4e4c214ec52ce76c59d8d06d56b52cbe2c10a: Status 404 returned error can't find the container with id 191610b24a51bf60e1910e1c30f4e4c214ec52ce76c59d8d06d56b52cbe2c10a
Nov 02 15:45:13.373839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.373804    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:13.373946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.373893    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:13.373946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.373941    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:13.562055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.562020    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:13.764372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.763931    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" event={"ID":"02302e75-aacc-4653-b806-4f06991d24c3","Type":"ContainerStarted","Data":"c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d"}
Nov 02 15:45:13.764372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.764004    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" event={"ID":"02302e75-aacc-4653-b806-4f06991d24c3","Type":"ContainerStarted","Data":"191610b24a51bf60e1910e1c30f4e4c214ec52ce76c59d8d06d56b52cbe2c10a"}
Nov 02 15:45:13.820069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.820026    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 15:45:13.820284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.820102    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 15:45:13.823393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:13.823347    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podStartSLOduration=1.823327338 podStartE2EDuration="1.823327338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:13.818994162 +0000 UTC m=+2908.072947786" watchObservedRunningTime="2026-11-02 15:45:13.823327338 +0000 UTC m=+2908.077280961"
Nov 02 15:45:14.003674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:14.002966    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:14.022725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:14.017794    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:45:14.022725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:14.017870    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Nov 02 15:45:14.136140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:14.136103    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:14.817993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:14.817951    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 15:45:14.818513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:14.818013    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 15:45:15.006243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:15.006211    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:15.803669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:15.803274    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:15.804445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:15.804421    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:45:15.817608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:15.817579    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 15:45:15.817764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:15.817635    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 15:45:15.928943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:15.928752    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:16.497921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:16.497883    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:16.695251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:16.695209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:16.816945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:16.816864    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 15:45:16.816945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:16.816916    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 15:45:17.817118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:17.817086    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 15:45:17.817698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:17.817626    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 15:45:18.302687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:18.302527    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:18.508984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:18.508249    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:18.817280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:18.817246    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body=
Nov 02 15:45:18.874141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:18.863749    7019 scope.go:122] "RemoveContainer" containerID="b484663d301f6e1274d3afe2436884691da8483a9756d88bb15ef9c60336c959"
Nov 02 15:45:18.890473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:18.890424    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused"
Nov 02 15:45:19.126231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.126168    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:19.589413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.589378    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"]
Nov 02 15:45:19.595554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.595530    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.614867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.614842    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-453864-openshift-default-dockercfg-nvg5q\""
Nov 02 15:45:19.618206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.618175    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"]
Nov 02 15:45:19.723292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723260    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-data\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723318    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723438    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-podinfo\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723477    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-credential-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723552    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-certs\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723641    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-token\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723706    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-envoy\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723779    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b85cg\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-kube-api-access-b85cg\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.723941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.723926    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istiod-ca-cert\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.822611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.822578    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:19.823093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.823078    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:19.823794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.823776    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:45:19.825571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825552    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-data\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.825639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825600    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.825714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825699    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-podinfo\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.825756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825736    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-credential-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.825809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825778    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-certs\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.825863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825818    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-token\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.825863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825848    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-envoy\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.825973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.825890    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b85cg\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-kube-api-access-b85cg\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.826084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.826043    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.826152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.826134    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-data\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.826219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.826193    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-credential-socket\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.826277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.826227    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istiod-ca-cert\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.826277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.826245    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-certs\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.826904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.826881    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istiod-ca-cert\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.828456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.828436    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-envoy\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.828880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.828864    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-podinfo\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.855203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.855175    7019 scope.go:122] "RemoveContainer" containerID="0d9d08aa178b46e07fdd7b1c43b4e352c3a4404786d8c94ff83f12a71bd87863"
Nov 02 15:45:19.855409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.855384    7019 scope.go:122] "RemoveContainer" containerID="cf061d60252cd15291e3d3c9d8f0595ba4d50c14d7b9edf647562c37ead02ccd"
Nov 02 15:45:19.855574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.855552    7019 scope.go:122] "RemoveContainer" containerID="8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818"
Nov 02 15:45:19.855705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.855640    7019 scope.go:122] "RemoveContainer" containerID="d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37"
Nov 02 15:45:19.856076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.856013    7019 scope.go:122] "RemoveContainer" containerID="1f419190702c3b641a82c8335a93c69a495c177b36739ccc5b62b36cbaf4f6f9"
Nov 02 15:45:19.858352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.858274    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-token\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.858934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.858906    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b85cg\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-kube-api-access-b85cg\") pod \"e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") " pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.913364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.913337    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:19.956209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:19.956153    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerStarted","Data":"46a61bf057217e867fef6fa856c34178648a8e0ce525f746eeab148c546cbad3"}
Nov 02 15:45:20.419492 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:20.419463    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf51c40_d05d_4778_ac8e_5b9de73d775c.slice/crio-a4c3727fae8b3a9d77768a4141cdf188d35a1ea7974a71d0063817a55233b476 WatchSource:0}: Error finding container a4c3727fae8b3a9d77768a4141cdf188d35a1ea7974a71d0063817a55233b476: Status 404 returned error can't find the container with id a4c3727fae8b3a9d77768a4141cdf188d35a1ea7974a71d0063817a55233b476
Nov 02 15:45:20.422614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.422577    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:20.422724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.422670    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:20.422724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.422705    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:20.423667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.423630    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"]
Nov 02 15:45:20.855798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.855628    7019 scope.go:122] "RemoveContainer" containerID="115c5dda4059462efc4933808d51ff289ec28bab1438c24c83a1721937deb534"
Nov 02 15:45:20.856246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.856045    7019 scope.go:122] "RemoveContainer" containerID="6fe7ce6949aa9b13b3d08e05430ac73cfb658292869e478e82bbaa3002f83d14"
Nov 02 15:45:20.856246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.856237    7019 scope.go:122] "RemoveContainer" containerID="21a651b3f1b010856796204dc093b56ec0b16eca5bff5be394c7de89a0dbedc6"
Nov 02 15:45:20.970693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.968168    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" event={"ID":"ccf51c40-d05d-4778-ac8e-5b9de73d775c","Type":"ContainerStarted","Data":"bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5"}
Nov 02 15:45:20.970693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.968241    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" event={"ID":"ccf51c40-d05d-4778-ac8e-5b9de73d775c","Type":"ContainerStarted","Data":"a4c3727fae8b3a9d77768a4141cdf188d35a1ea7974a71d0063817a55233b476"}
Nov 02 15:45:20.972896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.972862    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" event={"ID":"b568d3ce-7e0d-4909-8835-f2efc6bd4793","Type":"ContainerStarted","Data":"e2269689367fe46949d73a57f761daf57830af3f3c27d1c67b65ea55bd209531"}
Nov 02 15:45:20.977713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.976243    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" event={"ID":"08cc86fb-8725-48e1-9a73-7ad63bc05755","Type":"ContainerStarted","Data":"4627c4a66e0e94cd5b806f68dda69604a337339b49ac01f5502b68244d5e8eea"}
Nov 02 15:45:20.986829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.986760    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerStarted","Data":"901ccf6ea5d36d982796d455ff605c6719a4bd3292424c8a1045ca2e53bcddb1"}
Nov 02 15:45:20.994401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.993252    7019 generic.go:356] "Generic (PLEG): container finished" podID="fa7b87a4-682a-4462-9610-63ee828259e3" containerID="46a61bf057217e867fef6fa856c34178648a8e0ce525f746eeab148c546cbad3" exitCode=60
Nov 02 15:45:20.994401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.993339    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerDied","Data":"46a61bf057217e867fef6fa856c34178648a8e0ce525f746eeab148c546cbad3"}
Nov 02 15:45:20.994401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.993382    7019 scope.go:122] "RemoveContainer" containerID="b484663d301f6e1274d3afe2436884691da8483a9756d88bb15ef9c60336c959"
Nov 02 15:45:20.994401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.993790    7019 scope.go:122] "RemoveContainer" containerID="46a61bf057217e867fef6fa856c34178648a8e0ce525f746eeab148c546cbad3"
Nov 02 15:45:20.994737 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:20.994719    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z_odh-ai-gateway-infra(fa7b87a4-682a-4462-9610-63ee828259e3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" podUID="fa7b87a4-682a-4462-9610-63ee828259e3"
Nov 02 15:45:21.000279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:20.999625    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerStarted","Data":"63ba229edc14d0af7be8580b4f7194135c0451f19e60caaaa46afcd106a0f65b"}
Nov 02 15:45:21.008596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.008565    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerStarted","Data":"73641e01be7c39fc7bb6292b273195fb0261a0958322325cc072b8b93b5115d8"}
Nov 02 15:45:21.040517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.038602    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podStartSLOduration=2.038583317 podStartE2EDuration="2.038583317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:21.034468439 +0000 UTC m=+2915.288422063" watchObservedRunningTime="2026-11-02 15:45:21.038583317 +0000 UTC m=+2915.292536942"
Nov 02 15:45:21.042149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.042111    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:21.062176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.062142    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"]
Nov 02 15:45:21.141434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.141379    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" podStartSLOduration=21.141365849 podStartE2EDuration="21.141365849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:21.137901625 +0000 UTC m=+2915.391855252" watchObservedRunningTime="2026-11-02 15:45:21.141365849 +0000 UTC m=+2915.395319473"
Nov 02 15:45:21.855675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.855553    7019 scope.go:122] "RemoveContainer" containerID="695812f8650a4fd0a9c96f022b3eb92cddc9d792d2d771a6561975408c3cf201"
Nov 02 15:45:21.855856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.855742    7019 scope.go:122] "RemoveContainer" containerID="60989db43c764905b0d9799c95c08412a1178d6065e3c5c50d7fcb6c99f47c69"
Nov 02 15:45:21.856152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.856037    7019 scope.go:122] "RemoveContainer" containerID="f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23"
Nov 02 15:45:21.856325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.856298    7019 scope.go:122] "RemoveContainer" containerID="1bb6f670d4ac7f938cae9c9b64be9f461b4b67618f7aca3ac8945fe4e0b6ed35"
Nov 02 15:45:21.914281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.914240    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:21.915543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.915497    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Nov 02 15:45:21.915707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:21.915543    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Nov 02 15:45:22.030332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.030304    7019 generic.go:356] "Generic (PLEG): container finished" podID="935acbcf-5b16-4096-8a48-73579d2b6505" containerID="73641e01be7c39fc7bb6292b273195fb0261a0958322325cc072b8b93b5115d8" exitCode=60
Nov 02 15:45:22.030522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.030448    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerDied","Data":"73641e01be7c39fc7bb6292b273195fb0261a0958322325cc072b8b93b5115d8"}
Nov 02 15:45:22.030616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.030568    7019 scope.go:122] "RemoveContainer" containerID="1f419190702c3b641a82c8335a93c69a495c177b36739ccc5b62b36cbaf4f6f9"
Nov 02 15:45:22.031150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.031125    7019 scope.go:122] "RemoveContainer" containerID="73641e01be7c39fc7bb6292b273195fb0261a0958322325cc072b8b93b5115d8"
Nov 02 15:45:22.031553 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:22.031523    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z_odh-ai-gateway-infra(935acbcf-5b16-4096-8a48-73579d2b6505)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" podUID="935acbcf-5b16-4096-8a48-73579d2b6505"
Nov 02 15:45:22.033637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.033618    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerStarted","Data":"267dcac68e9bd9cdf2101d3bb8dcc78bc58d8a2a8ce94dab5273ac99117f2bbd"}
Nov 02 15:45:22.036456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.036435    7019 generic.go:356] "Generic (PLEG): container finished" podID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerID="e2269689367fe46949d73a57f761daf57830af3f3c27d1c67b65ea55bd209531" exitCode=60
Nov 02 15:45:22.036544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.036515    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" event={"ID":"b568d3ce-7e0d-4909-8835-f2efc6bd4793","Type":"ContainerDied","Data":"e2269689367fe46949d73a57f761daf57830af3f3c27d1c67b65ea55bd209531"}
Nov 02 15:45:22.036887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.036872    7019 scope.go:122] "RemoveContainer" containerID="e2269689367fe46949d73a57f761daf57830af3f3c27d1c67b65ea55bd209531"
Nov 02 15:45:22.037203 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:22.037180    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt_odh-ai-gateway-infra(b568d3ce-7e0d-4909-8835-f2efc6bd4793)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793"
Nov 02 15:45:22.039461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.039432    7019 generic.go:356] "Generic (PLEG): container finished" podID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerID="4627c4a66e0e94cd5b806f68dda69604a337339b49ac01f5502b68244d5e8eea" exitCode=60
Nov 02 15:45:22.039554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.039539    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" event={"ID":"08cc86fb-8725-48e1-9a73-7ad63bc05755","Type":"ContainerDied","Data":"4627c4a66e0e94cd5b806f68dda69604a337339b49ac01f5502b68244d5e8eea"}
Nov 02 15:45:22.039985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.039954    7019 scope.go:122] "RemoveContainer" containerID="4627c4a66e0e94cd5b806f68dda69604a337339b49ac01f5502b68244d5e8eea"
Nov 02 15:45:22.040380 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:22.040356    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds_odh-ai-gateway-infra(08cc86fb-8725-48e1-9a73-7ad63bc05755)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755"
Nov 02 15:45:22.043587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.043563    7019 generic.go:356] "Generic (PLEG): container finished" podID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerID="901ccf6ea5d36d982796d455ff605c6719a4bd3292424c8a1045ca2e53bcddb1" exitCode=60
Nov 02 15:45:22.043787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.043763    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerDied","Data":"901ccf6ea5d36d982796d455ff605c6719a4bd3292424c8a1045ca2e53bcddb1"}
Nov 02 15:45:22.044150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.044131    7019 scope.go:122] "RemoveContainer" containerID="901ccf6ea5d36d982796d455ff605c6719a4bd3292424c8a1045ca2e53bcddb1"
Nov 02 15:45:22.044394 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:22.044373    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s_odh-ai-gateway-infra(3d681fe3-a40e-411a-9a80-8b2c7b90a9d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9"
Nov 02 15:45:22.048212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.048186    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" event={"ID":"98247b5f-4631-4cbe-9fc7-a447913966b7","Type":"ContainerStarted","Data":"2e5b81d23cb9b84bf6fedd92404eddafd3c10444fffc299bee57e6bce73fc86c"}
Nov 02 15:45:22.050977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.050949    7019 generic.go:356] "Generic (PLEG): container finished" podID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerID="63ba229edc14d0af7be8580b4f7194135c0451f19e60caaaa46afcd106a0f65b" exitCode=60
Nov 02 15:45:22.051073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.051055    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerDied","Data":"63ba229edc14d0af7be8580b4f7194135c0451f19e60caaaa46afcd106a0f65b"}
Nov 02 15:45:22.051423 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.051399    7019 scope.go:122] "RemoveContainer" containerID="63ba229edc14d0af7be8580b4f7194135c0451f19e60caaaa46afcd106a0f65b"
Nov 02 15:45:22.051764 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:22.051737    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t_odh-ai-gateway-infra(2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b"
Nov 02 15:45:22.053769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.053748    7019 scope.go:122] "RemoveContainer" containerID="cf061d60252cd15291e3d3c9d8f0595ba4d50c14d7b9edf647562c37ead02ccd"
Nov 02 15:45:22.054489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.054466    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" event={"ID":"75fa808b-30f1-4090-859a-c8bb8fd9a999","Type":"ContainerStarted","Data":"7c2a71aa0747fccc8690940da7cde8019dab13b02eb4ec968ecaed109af6956d"}
Nov 02 15:45:22.128412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.128381    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"]
Nov 02 15:45:22.160994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.160963    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"]
Nov 02 15:45:22.230495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.230464    7019 scope.go:122] "RemoveContainer" containerID="0d9d08aa178b46e07fdd7b1c43b4e352c3a4404786d8c94ff83f12a71bd87863"
Nov 02 15:45:22.237940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.237909    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"]
Nov 02 15:45:22.247367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.247182    7019 scope.go:122] "RemoveContainer" containerID="d9a5cc7dd3b16a135b9745598b8899e0325a6218d041d535dde62c1ff2d79b37"
Nov 02 15:45:22.322924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.318974    7019 scope.go:122] "RemoveContainer" containerID="8650c3965f7b4f34f71140a2996a43bca8fb3d78a67e2457cf26c53c5907d818"
Nov 02 15:45:22.328621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.328594    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"]
Nov 02 15:45:22.350519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.348624    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"]
Nov 02 15:45:22.562499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.562476    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:22.634282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.634254    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7b87a4-682a-4462-9610-63ee828259e3-kube-api-access-grvwz\" (UniqueName: \"kubernetes.io/projected/fa7b87a4-682a-4462-9610-63ee828259e3-kube-api-access-grvwz\") pod \"fa7b87a4-682a-4462-9610-63ee828259e3\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") "
Nov 02 15:45:22.634520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.634505    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7b87a4-682a-4462-9610-63ee828259e3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b87a4-682a-4462-9610-63ee828259e3-maas-api-service-ca\") pod \"fa7b87a4-682a-4462-9610-63ee828259e3\" (UID: \"fa7b87a4-682a-4462-9610-63ee828259e3\") "
Nov 02 15:45:22.634900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.634879    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7b87a4-682a-4462-9610-63ee828259e3-maas-api-service-ca" pod "fa7b87a4-682a-4462-9610-63ee828259e3" (UID: "fa7b87a4-682a-4462-9610-63ee828259e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:22.635371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.635353    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa7b87a4-682a-4462-9610-63ee828259e3-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:22.636853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.636742    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7b87a4-682a-4462-9610-63ee828259e3-kube-api-access-grvwz" pod "fa7b87a4-682a-4462-9610-63ee828259e3" (UID: "fa7b87a4-682a-4462-9610-63ee828259e3"). InnerVolumeSpecName "kube-api-access-grvwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:22.740086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.739999    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grvwz\" (UniqueName: \"kubernetes.io/projected/fa7b87a4-682a-4462-9610-63ee828259e3-kube-api-access-grvwz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:22.805749 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:22.804855    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:45:22.914147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.914107    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Nov 02 15:45:22.914586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:22.914174    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Nov 02 15:45:23.057261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.057150    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:23.085159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.069433    7019 generic.go:356] "Generic (PLEG): container finished" podID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerID="7c2a71aa0747fccc8690940da7cde8019dab13b02eb4ec968ecaed109af6956d" exitCode=60
Nov 02 15:45:23.085159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.069562    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" event={"ID":"75fa808b-30f1-4090-859a-c8bb8fd9a999","Type":"ContainerDied","Data":"7c2a71aa0747fccc8690940da7cde8019dab13b02eb4ec968ecaed109af6956d"}
Nov 02 15:45:23.085159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.069624    7019 scope.go:122] "RemoveContainer" containerID="115c5dda4059462efc4933808d51ff289ec28bab1438c24c83a1721937deb534"
Nov 02 15:45:23.085159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.070176    7019 scope.go:122] "RemoveContainer" containerID="7c2a71aa0747fccc8690940da7cde8019dab13b02eb4ec968ecaed109af6956d"
Nov 02 15:45:23.085159 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:23.070602    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55_odh-ai-gateway-infra(75fa808b-30f1-4090-859a-c8bb8fd9a999)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999"
Nov 02 15:45:23.110371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.109586    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerStarted","Data":"6d2a0b78dc0bc515b7280fda0637a669d3e20981fa59a934ea46d105d5f0bd35"}
Nov 02 15:45:23.114230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.114176    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerStarted","Data":"05747c2d462b1857e9dcbfb1c3ff9ea762ff7d99d69c4637228808de8bc4169c"}
Nov 02 15:45:23.118194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.118166    7019 generic.go:356] "Generic (PLEG): container finished" podID="73bca953-9136-42bd-916d-1e3f1915b53a" containerID="267dcac68e9bd9cdf2101d3bb8dcc78bc58d8a2a8ce94dab5273ac99117f2bbd" exitCode=60
Nov 02 15:45:23.118418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.118236    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerDied","Data":"267dcac68e9bd9cdf2101d3bb8dcc78bc58d8a2a8ce94dab5273ac99117f2bbd"}
Nov 02 15:45:23.118932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.118914    7019 scope.go:122] "RemoveContainer" containerID="267dcac68e9bd9cdf2101d3bb8dcc78bc58d8a2a8ce94dab5273ac99117f2bbd"
Nov 02 15:45:23.119454 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:23.119429    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp_odh-ai-gateway-infra(73bca953-9136-42bd-916d-1e3f1915b53a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" podUID="73bca953-9136-42bd-916d-1e3f1915b53a"
Nov 02 15:45:23.128955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.128676    7019 scope.go:122] "RemoveContainer" containerID="21a651b3f1b010856796204dc093b56ec0b16eca5bff5be394c7de89a0dbedc6"
Nov 02 15:45:23.136148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.136082    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerStarted","Data":"87c06f70c6bf6b54f4eb0f77a1448637809ff927e63594756a94979d246ed58a"}
Nov 02 15:45:23.140667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.140504    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z" event={"ID":"fa7b87a4-682a-4462-9610-63ee828259e3","Type":"ContainerDied","Data":"42841d0b481840c5516ec7144ffcba4f20353113d882cbc42225a13616899848"}
Nov 02 15:45:23.142521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.142496    7019 generic.go:356] "Generic (PLEG): container finished" podID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerID="2e5b81d23cb9b84bf6fedd92404eddafd3c10444fffc299bee57e6bce73fc86c" exitCode=60
Nov 02 15:45:23.142616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.142575    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" event={"ID":"98247b5f-4631-4cbe-9fc7-a447913966b7","Type":"ContainerDied","Data":"2e5b81d23cb9b84bf6fedd92404eddafd3c10444fffc299bee57e6bce73fc86c"}
Nov 02 15:45:23.142961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.142942    7019 scope.go:122] "RemoveContainer" containerID="2e5b81d23cb9b84bf6fedd92404eddafd3c10444fffc299bee57e6bce73fc86c"
Nov 02 15:45:23.143277 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:23.143253    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r_odh-ai-gateway-infra(98247b5f-4631-4cbe-9fc7-a447913966b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7"
Nov 02 15:45:23.144991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.144969    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" event={"ID":"e4c120c3-da25-4a39-8a28-916dce067311","Type":"ContainerStarted","Data":"eb58f07b1e62758335feb0cacd37cb18856473c7aab81e67cc56f0615e88dfdb"}
Nov 02 15:45:23.145728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.145708    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"
Nov 02 15:45:23.152966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.152771    7019 scope.go:122] "RemoveContainer" containerID="46a61bf057217e867fef6fa856c34178648a8e0ce525f746eeab148c546cbad3"
Nov 02 15:45:23.224713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.224505    7019 scope.go:122] "RemoveContainer" containerID="6fe7ce6949aa9b13b3d08e05430ac73cfb658292869e478e82bbaa3002f83d14"
Nov 02 15:45:23.235778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.235747    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"]
Nov 02 15:45:23.517448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.517356    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"]
Nov 02 15:45:23.519900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.519851    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:23.554530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.554508    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:23.557618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.557440    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935acbcf-5b16-4096-8a48-73579d2b6505-kube-api-access-ht2r6\" (UniqueName: \"kubernetes.io/projected/935acbcf-5b16-4096-8a48-73579d2b6505-kube-api-access-ht2r6\") pod \"935acbcf-5b16-4096-8a48-73579d2b6505\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") "
Nov 02 15:45:23.557618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.557509    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935acbcf-5b16-4096-8a48-73579d2b6505-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935acbcf-5b16-4096-8a48-73579d2b6505-maas-api-service-ca\") pod \"935acbcf-5b16-4096-8a48-73579d2b6505\" (UID: \"935acbcf-5b16-4096-8a48-73579d2b6505\") "
Nov 02 15:45:23.557922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.557888    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935acbcf-5b16-4096-8a48-73579d2b6505-maas-api-service-ca" pod "935acbcf-5b16-4096-8a48-73579d2b6505" (UID: "935acbcf-5b16-4096-8a48-73579d2b6505"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.558355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.558302    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/935acbcf-5b16-4096-8a48-73579d2b6505-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.561900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.560673    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935acbcf-5b16-4096-8a48-73579d2b6505-kube-api-access-ht2r6" pod "935acbcf-5b16-4096-8a48-73579d2b6505" (UID: "935acbcf-5b16-4096-8a48-73579d2b6505"). InnerVolumeSpecName "kube-api-access-ht2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.659736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.659706    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-kube-api-access-vdfdd\" (UniqueName: \"kubernetes.io/projected/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-kube-api-access-vdfdd\") pod \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") "
Nov 02 15:45:23.660582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.660560    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-maas-api-service-ca\") pod \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\" (UID: \"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9\") "
Nov 02 15:45:23.661283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.661267    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2r6\" (UniqueName: \"kubernetes.io/projected/935acbcf-5b16-4096-8a48-73579d2b6505-kube-api-access-ht2r6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.661697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.661643    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-maas-api-service-ca" pod "3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" (UID: "3d681fe3-a40e-411a-9a80-8b2c7b90a9d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.663624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.663599    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"]
Nov 02 15:45:23.667763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.665084    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-kube-api-access-vdfdd" pod "3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" (UID: "3d681fe3-a40e-411a-9a80-8b2c7b90a9d9"). InnerVolumeSpecName "kube-api-access-vdfdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.670389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.670329    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-7ecfe1-29893905-gdz8z"]
Nov 02 15:45:23.678759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.678736    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"]
Nov 02 15:45:23.763252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.763068    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:23.792790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.772733    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfdd\" (UniqueName: \"kubernetes.io/projected/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-kube-api-access-vdfdd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.792790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.772764    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.818009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.817930    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:23.820212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.820181    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:23.891773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.884127    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b568d3ce-7e0d-4909-8835-f2efc6bd4793-kube-api-access-sbmcj\" (UniqueName: \"kubernetes.io/projected/b568d3ce-7e0d-4909-8835-f2efc6bd4793-kube-api-access-sbmcj\") pod \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") "
Nov 02 15:45:23.891773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.884239    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08cc86fb-8725-48e1-9a73-7ad63bc05755-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cc86fb-8725-48e1-9a73-7ad63bc05755-maas-api-service-ca\") pod \"08cc86fb-8725-48e1-9a73-7ad63bc05755\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") "
Nov 02 15:45:23.891773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.884440    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-kube-api-access-xf8cr\" (UniqueName: \"kubernetes.io/projected/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-kube-api-access-xf8cr\") pod \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") "
Nov 02 15:45:23.891773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.884473    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b568d3ce-7e0d-4909-8835-f2efc6bd4793-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b568d3ce-7e0d-4909-8835-f2efc6bd4793-maas-api-service-ca\") pod \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\" (UID: \"b568d3ce-7e0d-4909-8835-f2efc6bd4793\") "
Nov 02 15:45:23.891773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.884523    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08cc86fb-8725-48e1-9a73-7ad63bc05755-kube-api-access-zpmbc\" (UniqueName: \"kubernetes.io/projected/08cc86fb-8725-48e1-9a73-7ad63bc05755-kube-api-access-zpmbc\") pod \"08cc86fb-8725-48e1-9a73-7ad63bc05755\" (UID: \"08cc86fb-8725-48e1-9a73-7ad63bc05755\") "
Nov 02 15:45:23.891773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.884636    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-maas-api-service-ca\") pod \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\" (UID: \"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b\") "
Nov 02 15:45:23.900667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.896180    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-maas-api-service-ca" pod "2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" (UID: "2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.900667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.897673    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b568d3ce-7e0d-4909-8835-f2efc6bd4793-maas-api-service-ca" pod "b568d3ce-7e0d-4909-8835-f2efc6bd4793" (UID: "b568d3ce-7e0d-4909-8835-f2efc6bd4793"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.903679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.901811    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08cc86fb-8725-48e1-9a73-7ad63bc05755-maas-api-service-ca" pod "08cc86fb-8725-48e1-9a73-7ad63bc05755" (UID: "08cc86fb-8725-48e1-9a73-7ad63bc05755"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:23.911192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.911150    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08cc86fb-8725-48e1-9a73-7ad63bc05755-kube-api-access-zpmbc" pod "08cc86fb-8725-48e1-9a73-7ad63bc05755" (UID: "08cc86fb-8725-48e1-9a73-7ad63bc05755"). InnerVolumeSpecName "kube-api-access-zpmbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.912926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.912811    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-kube-api-access-xf8cr" pod "2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" (UID: "2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b"). InnerVolumeSpecName "kube-api-access-xf8cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.912926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.912890    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b568d3ce-7e0d-4909-8835-f2efc6bd4793-kube-api-access-sbmcj" pod "b568d3ce-7e0d-4909-8835-f2efc6bd4793" (UID: "b568d3ce-7e0d-4909-8835-f2efc6bd4793"). InnerVolumeSpecName "kube-api-access-sbmcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:23.918717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.913424    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" path="/var/lib/kubelet/pods/fa7b87a4-682a-4462-9610-63ee828259e3/volumes"
Nov 02 15:45:23.918717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.918441    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Nov 02 15:45:23.918717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.918485    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Nov 02 15:45:23.998799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.993341    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf8cr\" (UniqueName: \"kubernetes.io/projected/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-kube-api-access-xf8cr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.998799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.993373    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b568d3ce-7e0d-4909-8835-f2efc6bd4793-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.998799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.993390    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpmbc\" (UniqueName: \"kubernetes.io/projected/08cc86fb-8725-48e1-9a73-7ad63bc05755-kube-api-access-zpmbc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.998799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.993403    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.998799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.993421    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmcj\" (UniqueName: \"kubernetes.io/projected/b568d3ce-7e0d-4909-8835-f2efc6bd4793-kube-api-access-sbmcj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:23.998799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:23.993435    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08cc86fb-8725-48e1-9a73-7ad63bc05755-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.229159 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.229122    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds" event={"ID":"08cc86fb-8725-48e1-9a73-7ad63bc05755","Type":"ContainerDied","Data":"aa16a546dcd84775a6501a7257f6022e412144170e8f08dc034413af5424163f"}
Nov 02 15:45:24.229374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.229209    7019 scope.go:122] "RemoveContainer" containerID="4627c4a66e0e94cd5b806f68dda69604a337339b49ac01f5502b68244d5e8eea"
Nov 02 15:45:24.229374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.229306    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"
Nov 02 15:45:24.236662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.236614    7019 generic.go:356] "Generic (PLEG): container finished" podID="92ed44b3-3268-4076-a279-829f1c7facaf" containerID="87c06f70c6bf6b54f4eb0f77a1448637809ff927e63594756a94979d246ed58a" exitCode=60
Nov 02 15:45:24.236808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.236685    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerDied","Data":"87c06f70c6bf6b54f4eb0f77a1448637809ff927e63594756a94979d246ed58a"}
Nov 02 15:45:24.238401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.238377    7019 scope.go:122] "RemoveContainer" containerID="87c06f70c6bf6b54f4eb0f77a1448637809ff927e63594756a94979d246ed58a"
Nov 02 15:45:24.238829 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:24.238801    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6_odh-ai-gateway-infra(92ed44b3-3268-4076-a279-829f1c7facaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" podUID="92ed44b3-3268-4076-a279-829f1c7facaf"
Nov 02 15:45:24.241554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.241508    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s" event={"ID":"3d681fe3-a40e-411a-9a80-8b2c7b90a9d9","Type":"ContainerDied","Data":"9f79c877f9e425934ca742fd4e728e81f8c573b20ede9d2915567e793776ab6d"}
Nov 02 15:45:24.241702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.241682    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"
Nov 02 15:45:24.249980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.249946    7019 scope.go:122] "RemoveContainer" containerID="f47721293fbcedf060eb03dfa5a73dac1401fe51dd7ad05b7bcb7e78b2d14c23"
Nov 02 15:45:24.253180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.253152    7019 generic.go:356] "Generic (PLEG): container finished" podID="e4c120c3-da25-4a39-8a28-916dce067311" containerID="eb58f07b1e62758335feb0cacd37cb18856473c7aab81e67cc56f0615e88dfdb" exitCode=60
Nov 02 15:45:24.253298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.253261    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" event={"ID":"e4c120c3-da25-4a39-8a28-916dce067311","Type":"ContainerDied","Data":"eb58f07b1e62758335feb0cacd37cb18856473c7aab81e67cc56f0615e88dfdb"}
Nov 02 15:45:24.254214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.254190    7019 scope.go:122] "RemoveContainer" containerID="eb58f07b1e62758335feb0cacd37cb18856473c7aab81e67cc56f0615e88dfdb"
Nov 02 15:45:24.254657 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:24.254607    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5_odh-ai-gateway-infra(e4c120c3-da25-4a39-8a28-916dce067311)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" podUID="e4c120c3-da25-4a39-8a28-916dce067311"
Nov 02 15:45:24.256081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.256053    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t" event={"ID":"2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b","Type":"ContainerDied","Data":"a9c7974813522f9f92bc205e4369e1c2bc3e118be62be6be4523313edf5524fc"}
Nov 02 15:45:24.256197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.256163    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"
Nov 02 15:45:24.264207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.264091    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"]
Nov 02 15:45:24.277052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.276789    7019 scope.go:122] "RemoveContainer" containerID="901ccf6ea5d36d982796d455ff605c6719a4bd3292424c8a1045ca2e53bcddb1"
Nov 02 15:45:24.281816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.280009    7019 generic.go:356] "Generic (PLEG): container finished" podID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerID="6d2a0b78dc0bc515b7280fda0637a669d3e20981fa59a934ea46d105d5f0bd35" exitCode=60
Nov 02 15:45:24.281816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.280101    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerDied","Data":"6d2a0b78dc0bc515b7280fda0637a669d3e20981fa59a934ea46d105d5f0bd35"}
Nov 02 15:45:24.281816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.280731    7019 scope.go:122] "RemoveContainer" containerID="6d2a0b78dc0bc515b7280fda0637a669d3e20981fa59a934ea46d105d5f0bd35"
Nov 02 15:45:24.281816 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:24.281188    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt_odh-ai-gateway-infra(1a625702-a861-4efb-9bf5-b50c4f3119cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf"
Nov 02 15:45:24.334793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.333762    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"]
Nov 02 15:45:24.337248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.335362    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z" event={"ID":"935acbcf-5b16-4096-8a48-73579d2b6505","Type":"ContainerDied","Data":"1773687e6c84cc0b0b5e21326936d6db3c82f7e31f22d2b4d93be5814147c5b2"}
Nov 02 15:45:24.337248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.335484    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"
Nov 02 15:45:24.337248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.336935    7019 scope.go:122] "RemoveContainer" containerID="695812f8650a4fd0a9c96f022b3eb92cddc9d792d2d771a6561975408c3cf201"
Nov 02 15:45:24.338444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.338370    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-739bf1-29893905-867ds"]
Nov 02 15:45:24.340980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.340959    7019 generic.go:356] "Generic (PLEG): container finished" podID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerID="05747c2d462b1857e9dcbfb1c3ff9ea762ff7d99d69c4637228808de8bc4169c" exitCode=60
Nov 02 15:45:24.341308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.341196    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerDied","Data":"05747c2d462b1857e9dcbfb1c3ff9ea762ff7d99d69c4637228808de8bc4169c"}
Nov 02 15:45:24.347888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.347871    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt" event={"ID":"b568d3ce-7e0d-4909-8835-f2efc6bd4793","Type":"ContainerDied","Data":"9018dc3b7f9497cddf32512d61bd313d04738d7ab18b6c097851c0272e210d39"}
Nov 02 15:45:24.348069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.348057    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"
Nov 02 15:45:24.363153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.363124    7019 scope.go:122] "RemoveContainer" containerID="63ba229edc14d0af7be8580b4f7194135c0451f19e60caaaa46afcd106a0f65b"
Nov 02 15:45:24.432670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.427875    7019 scope.go:122] "RemoveContainer" containerID="1bb6f670d4ac7f938cae9c9b64be9f461b4b67618f7aca3ac8945fe4e0b6ed35"
Nov 02 15:45:24.520553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.505760    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"]
Nov 02 15:45:24.520553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.512429    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-797488-29893905-dzf6s"]
Nov 02 15:45:24.536243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.536212    7019 scope.go:122] "RemoveContainer" containerID="73641e01be7c39fc7bb6292b273195fb0261a0958322325cc072b8b93b5115d8"
Nov 02 15:45:24.539854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.539803    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"]
Nov 02 15:45:24.628977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.628952    7019 scope.go:122] "RemoveContainer" containerID="60989db43c764905b0d9799c95c08412a1178d6065e3c5c50d7fcb6c99f47c69"
Nov 02 15:45:24.663169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.662235    7019 scope.go:122] "RemoveContainer" containerID="e2269689367fe46949d73a57f761daf57830af3f3c27d1c67b65ea55bd209531"
Nov 02 15:45:24.663470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.663392    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:24.716481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.716456    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"]
Nov 02 15:45:24.723422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.723393    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-e80c77-29893905-h9s6z"]
Nov 02 15:45:24.729274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.728780    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"]
Nov 02 15:45:24.743430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.742544    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75fa808b-30f1-4090-859a-c8bb8fd9a999-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75fa808b-30f1-4090-859a-c8bb8fd9a999-maas-api-service-ca\") pod \"75fa808b-30f1-4090-859a-c8bb8fd9a999\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") "
Nov 02 15:45:24.743430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.742600    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75fa808b-30f1-4090-859a-c8bb8fd9a999-kube-api-access-mxm22\" (UniqueName: \"kubernetes.io/projected/75fa808b-30f1-4090-859a-c8bb8fd9a999-kube-api-access-mxm22\") pod \"75fa808b-30f1-4090-859a-c8bb8fd9a999\" (UID: \"75fa808b-30f1-4090-859a-c8bb8fd9a999\") "
Nov 02 15:45:24.743430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.743054    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75fa808b-30f1-4090-859a-c8bb8fd9a999-maas-api-service-ca" pod "75fa808b-30f1-4090-859a-c8bb8fd9a999" (UID: "75fa808b-30f1-4090-859a-c8bb8fd9a999"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:24.743430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.743389    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75fa808b-30f1-4090-859a-c8bb8fd9a999-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.746464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.746225    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75fa808b-30f1-4090-859a-c8bb8fd9a999-kube-api-access-mxm22" pod "75fa808b-30f1-4090-859a-c8bb8fd9a999" (UID: "75fa808b-30f1-4090-859a-c8bb8fd9a999"). InnerVolumeSpecName "kube-api-access-mxm22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:24.837719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.834274    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"]
Nov 02 15:45:24.840075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.840014    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-9cfd1f-29893905-tfj5t"]
Nov 02 15:45:24.846414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.846383    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxm22\" (UniqueName: \"kubernetes.io/projected/75fa808b-30f1-4090-859a-c8bb8fd9a999-kube-api-access-mxm22\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.854445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.854425    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:24.860409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.860390    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:24.862539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.862519    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:24.920748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.920714    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"]
Nov 02 15:45:24.922828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.922096    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Nov 02 15:45:24.922828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.922811    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Nov 02 15:45:24.946517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.946477    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:24.946975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.946949    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98247b5f-4631-4cbe-9fc7-a447913966b7-kube-api-access-7tx5g\" (UniqueName: \"kubernetes.io/projected/98247b5f-4631-4cbe-9fc7-a447913966b7-kube-api-access-7tx5g\") pod \"98247b5f-4631-4cbe-9fc7-a447913966b7\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") "
Nov 02 15:45:24.947170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.947151    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98247b5f-4631-4cbe-9fc7-a447913966b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98247b5f-4631-4cbe-9fc7-a447913966b7-maas-api-service-ca\") pod \"98247b5f-4631-4cbe-9fc7-a447913966b7\" (UID: \"98247b5f-4631-4cbe-9fc7-a447913966b7\") "
Nov 02 15:45:24.947242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.947202    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-kube-api-access-c4qrd\" (UniqueName: \"kubernetes.io/projected/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-kube-api-access-c4qrd\") pod \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") "
Nov 02 15:45:24.947296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.947247    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-maas-api-service-ca\") pod \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\" (UID: \"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7\") "
Nov 02 15:45:24.947354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.947341    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bca953-9136-42bd-916d-1e3f1915b53a-kube-api-access-tdr9l\" (UniqueName: \"kubernetes.io/projected/73bca953-9136-42bd-916d-1e3f1915b53a-kube-api-access-tdr9l\") pod \"73bca953-9136-42bd-916d-1e3f1915b53a\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") "
Nov 02 15:45:24.947456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.947442    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bca953-9136-42bd-916d-1e3f1915b53a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bca953-9136-42bd-916d-1e3f1915b53a-maas-api-service-ca\") pod \"73bca953-9136-42bd-916d-1e3f1915b53a\" (UID: \"73bca953-9136-42bd-916d-1e3f1915b53a\") "
Nov 02 15:45:24.948545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.947546    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98247b5f-4631-4cbe-9fc7-a447913966b7-maas-api-service-ca" pod "98247b5f-4631-4cbe-9fc7-a447913966b7" (UID: "98247b5f-4631-4cbe-9fc7-a447913966b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:24.948545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.947745    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-maas-api-service-ca" pod "767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" (UID: "767a5bf4-2b1b-494d-bb0f-d9fca4e295d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:24.948545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.948152    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bca953-9136-42bd-916d-1e3f1915b53a-maas-api-service-ca" pod "73bca953-9136-42bd-916d-1e3f1915b53a" (UID: "73bca953-9136-42bd-916d-1e3f1915b53a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:24.948545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.948166    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.948545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.948183    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98247b5f-4631-4cbe-9fc7-a447913966b7-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:24.950144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.950121    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bca953-9136-42bd-916d-1e3f1915b53a-kube-api-access-tdr9l" pod "73bca953-9136-42bd-916d-1e3f1915b53a" (UID: "73bca953-9136-42bd-916d-1e3f1915b53a"). InnerVolumeSpecName "kube-api-access-tdr9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:24.950277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.950146    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-kube-api-access-c4qrd" pod "767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" (UID: "767a5bf4-2b1b-494d-bb0f-d9fca4e295d7"). InnerVolumeSpecName "kube-api-access-c4qrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:24.950407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:24.950184    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98247b5f-4631-4cbe-9fc7-a447913966b7-kube-api-access-7tx5g" pod "98247b5f-4631-4cbe-9fc7-a447913966b7" (UID: "98247b5f-4631-4cbe-9fc7-a447913966b7"). InnerVolumeSpecName "kube-api-access-7tx5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:25.041744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.041710    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"]
Nov 02 15:45:25.044922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.044895    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-b955c5-29893905-fx4xt"]
Nov 02 15:45:25.051274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.051246    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qrd\" (UniqueName: \"kubernetes.io/projected/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7-kube-api-access-c4qrd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:25.051402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.051275    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdr9l\" (UniqueName: \"kubernetes.io/projected/73bca953-9136-42bd-916d-1e3f1915b53a-kube-api-access-tdr9l\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:25.051402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.051292    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73bca953-9136-42bd-916d-1e3f1915b53a-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:25.051402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.051301    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tx5g\" (UniqueName: \"kubernetes.io/projected/98247b5f-4631-4cbe-9fc7-a447913966b7-kube-api-access-7tx5g\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:25.406670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.403712    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r" event={"ID":"98247b5f-4631-4cbe-9fc7-a447913966b7","Type":"ContainerDied","Data":"16842536fe8a9e43fea59e769ef34bdde9f0ca6ae5b7cdb9f5cb7ae7a7896995"}
Nov 02 15:45:25.406670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.403792    7019 scope.go:122] "RemoveContainer" containerID="2e5b81d23cb9b84bf6fedd92404eddafd3c10444fffc299bee57e6bce73fc86c"
Nov 02 15:45:25.406670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.403896    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"
Nov 02 15:45:25.442882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.442844    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55" event={"ID":"75fa808b-30f1-4090-859a-c8bb8fd9a999","Type":"ContainerDied","Data":"a56e24a6de7eb9c59ec893a6bb7e741a2c186cf89a976c7389d4ca2fb39ae00e"}
Nov 02 15:45:25.443048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.442960    7019 scope.go:122] "RemoveContainer" containerID="7c2a71aa0747fccc8690940da7cde8019dab13b02eb4ec968ecaed109af6956d"
Nov 02 15:45:25.443150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.443119    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"
Nov 02 15:45:25.456254 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.456220    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf" event={"ID":"767a5bf4-2b1b-494d-bb0f-d9fca4e295d7","Type":"ContainerDied","Data":"3713ec19eeb6cc3fd7e49c8478d2402b9399681d9189ce761e9835b63246bc4a"}
Nov 02 15:45:25.456529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.456509    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"
Nov 02 15:45:25.531803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.518971    7019 scope.go:122] "RemoveContainer" containerID="05747c2d462b1857e9dcbfb1c3ff9ea762ff7d99d69c4637228808de8bc4169c"
Nov 02 15:45:25.531803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.531016    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp" event={"ID":"73bca953-9136-42bd-916d-1e3f1915b53a","Type":"ContainerDied","Data":"b6232ad2e01ec60691222f974de329f2410739f156749cdef211463ff96d5909"}
Nov 02 15:45:25.536757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.531147    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"
Nov 02 15:45:25.552527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.552500    7019 scope.go:122] "RemoveContainer" containerID="267dcac68e9bd9cdf2101d3bb8dcc78bc58d8a2a8ce94dab5273ac99117f2bbd"
Nov 02 15:45:25.649136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.649058    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"]
Nov 02 15:45:25.654478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.653953    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-453864-29893905-vvr55"]
Nov 02 15:45:25.724141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.724115    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:25.736814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.736081    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"]
Nov 02 15:45:25.741557 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.741079    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-501738-29893905-6jm8r"]
Nov 02 15:45:25.778071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.765784    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ed44b3-3268-4076-a279-829f1c7facaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ed44b3-3268-4076-a279-829f1c7facaf-maas-api-service-ca\") pod \"92ed44b3-3268-4076-a279-829f1c7facaf\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") "
Nov 02 15:45:25.778071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.765845    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ed44b3-3268-4076-a279-829f1c7facaf-kube-api-access-z2r77\" (UniqueName: \"kubernetes.io/projected/92ed44b3-3268-4076-a279-829f1c7facaf-kube-api-access-z2r77\") pod \"92ed44b3-3268-4076-a279-829f1c7facaf\" (UID: \"92ed44b3-3268-4076-a279-829f1c7facaf\") "
Nov 02 15:45:25.778071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.767226    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ed44b3-3268-4076-a279-829f1c7facaf-maas-api-service-ca" pod "92ed44b3-3268-4076-a279-829f1c7facaf" (UID: "92ed44b3-3268-4076-a279-829f1c7facaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:25.819215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.819174    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ed44b3-3268-4076-a279-829f1c7facaf-kube-api-access-z2r77" pod "92ed44b3-3268-4076-a279-829f1c7facaf" (UID: "92ed44b3-3268-4076-a279-829f1c7facaf"). InnerVolumeSpecName "kube-api-access-z2r77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:25.859478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.859445    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"]
Nov 02 15:45:25.868949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.868074    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ed44b3-3268-4076-a279-829f1c7facaf-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:25.868949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.868104    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2r77\" (UniqueName: \"kubernetes.io/projected/92ed44b3-3268-4076-a279-829f1c7facaf-kube-api-access-z2r77\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:25.919196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.917997    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Nov 02 15:45:25.919196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.918054    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Nov 02 15:45:25.925321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.925274    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" path="/var/lib/kubelet/pods/08cc86fb-8725-48e1-9a73-7ad63bc05755/volumes"
Nov 02 15:45:25.926183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.926160    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" path="/var/lib/kubelet/pods/2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b/volumes"
Nov 02 15:45:25.926842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.926575    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:25.926842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.926823    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" path="/var/lib/kubelet/pods/3d681fe3-a40e-411a-9a80-8b2c7b90a9d9/volumes"
Nov 02 15:45:25.927862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.927843    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" path="/var/lib/kubelet/pods/75fa808b-30f1-4090-859a-c8bb8fd9a999/volumes"
Nov 02 15:45:25.928549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.928527    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" path="/var/lib/kubelet/pods/935acbcf-5b16-4096-8a48-73579d2b6505/volumes"
Nov 02 15:45:25.929151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.929135    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" path="/var/lib/kubelet/pods/98247b5f-4631-4cbe-9fc7-a447913966b7/volumes"
Nov 02 15:45:25.930016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.929998    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:25.930097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.930056    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" path="/var/lib/kubelet/pods/b568d3ce-7e0d-4909-8835-f2efc6bd4793/volumes"
Nov 02 15:45:25.930712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.930693    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8a3fb0-29893905-mj5pf"]
Nov 02 15:45:25.931125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.931112    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"]
Nov 02 15:45:25.938844 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:25.938827    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5dd8e9-29893905-54kcp"]
Nov 02 15:45:26.007021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.006826    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c120c3-da25-4a39-8a28-916dce067311-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c120c3-da25-4a39-8a28-916dce067311-maas-api-service-ca\") pod \"e4c120c3-da25-4a39-8a28-916dce067311\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") "
Nov 02 15:45:26.007984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.007310    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c120c3-da25-4a39-8a28-916dce067311-maas-api-service-ca" pod "e4c120c3-da25-4a39-8a28-916dce067311" (UID: "e4c120c3-da25-4a39-8a28-916dce067311"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:26.007984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.007318    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c120c3-da25-4a39-8a28-916dce067311-kube-api-access-46q4k\" (UniqueName: \"kubernetes.io/projected/e4c120c3-da25-4a39-8a28-916dce067311-kube-api-access-46q4k\") pod \"e4c120c3-da25-4a39-8a28-916dce067311\" (UID: \"e4c120c3-da25-4a39-8a28-916dce067311\") "
Nov 02 15:45:26.007984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.007431    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a625702-a861-4efb-9bf5-b50c4f3119cf-kube-api-access-bdcjt\" (UniqueName: \"kubernetes.io/projected/1a625702-a861-4efb-9bf5-b50c4f3119cf-kube-api-access-bdcjt\") pod \"1a625702-a861-4efb-9bf5-b50c4f3119cf\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") "
Nov 02 15:45:26.007984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.007716    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a625702-a861-4efb-9bf5-b50c4f3119cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a625702-a861-4efb-9bf5-b50c4f3119cf-maas-api-service-ca\") pod \"1a625702-a861-4efb-9bf5-b50c4f3119cf\" (UID: \"1a625702-a861-4efb-9bf5-b50c4f3119cf\") "
Nov 02 15:45:26.008904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.008708    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c120c3-da25-4a39-8a28-916dce067311-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.009491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.009129    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a625702-a861-4efb-9bf5-b50c4f3119cf-maas-api-service-ca" pod "1a625702-a861-4efb-9bf5-b50c4f3119cf" (UID: "1a625702-a861-4efb-9bf5-b50c4f3119cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:26.012827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.012320    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a625702-a861-4efb-9bf5-b50c4f3119cf-kube-api-access-bdcjt" pod "1a625702-a861-4efb-9bf5-b50c4f3119cf" (UID: "1a625702-a861-4efb-9bf5-b50c4f3119cf"). InnerVolumeSpecName "kube-api-access-bdcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:26.029504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.026571    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c120c3-da25-4a39-8a28-916dce067311-kube-api-access-46q4k" pod "e4c120c3-da25-4a39-8a28-916dce067311" (UID: "e4c120c3-da25-4a39-8a28-916dce067311"). InnerVolumeSpecName "kube-api-access-46q4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:26.111147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.111060    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46q4k\" (UniqueName: \"kubernetes.io/projected/e4c120c3-da25-4a39-8a28-916dce067311-kube-api-access-46q4k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.111147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.111103    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdcjt\" (UniqueName: \"kubernetes.io/projected/1a625702-a861-4efb-9bf5-b50c4f3119cf-kube-api-access-bdcjt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.111147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.111119    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a625702-a861-4efb-9bf5-b50c4f3119cf-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:26.544586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.544556    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"
Nov 02 15:45:26.544806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.544598    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt" event={"ID":"1a625702-a861-4efb-9bf5-b50c4f3119cf","Type":"ContainerDied","Data":"102d6de36408b031dc75cf458baa99df4b05d032d9f74bd3bde2e07df77c51f5"}
Nov 02 15:45:26.544806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.544708    7019 scope.go:122] "RemoveContainer" containerID="6d2a0b78dc0bc515b7280fda0637a669d3e20981fa59a934ea46d105d5f0bd35"
Nov 02 15:45:26.550613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.550210    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"
Nov 02 15:45:26.550613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.550235    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6" event={"ID":"92ed44b3-3268-4076-a279-829f1c7facaf","Type":"ContainerDied","Data":"e8a629a34bb1bcf0c6fee74aa8f93048def16a244e94324d749e012d870621b5"}
Nov 02 15:45:26.554144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.554114    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5" event={"ID":"e4c120c3-da25-4a39-8a28-916dce067311","Type":"ContainerDied","Data":"6bc4e1db1f12b996de20e9f61bd5e1dbc2b16b9a1aa6487f65f21914746d2359"}
Nov 02 15:45:26.554274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.554216    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"
Nov 02 15:45:26.589934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.587811    7019 scope.go:122] "RemoveContainer" containerID="87c06f70c6bf6b54f4eb0f77a1448637809ff927e63594756a94979d246ed58a"
Nov 02 15:45:26.633262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.631911    7019 scope.go:122] "RemoveContainer" containerID="eb58f07b1e62758335feb0cacd37cb18856473c7aab81e67cc56f0615e88dfdb"
Nov 02 15:45:26.684864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.684826    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"]
Nov 02 15:45:26.688682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.688630    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-3805b8-29893905-n5lr5"]
Nov 02 15:45:26.737674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.737620    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"]
Nov 02 15:45:26.744046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.744016    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f6f1a0-29893905-kwcm6"]
Nov 02 15:45:26.759903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.759869    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:26.802695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.802598    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"]
Nov 02 15:45:26.818744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.816943    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5dd139-29893905-sqcdt"]
Nov 02 15:45:26.913970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.913931    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body=
Nov 02 15:45:26.914140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:26.913993    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused"
Nov 02 15:45:27.864750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.864720    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" path="/var/lib/kubelet/pods/1a625702-a861-4efb-9bf5-b50c4f3119cf/volumes"
Nov 02 15:45:27.865224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.865210    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" path="/var/lib/kubelet/pods/73bca953-9136-42bd-916d-1e3f1915b53a/volumes"
Nov 02 15:45:27.865630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.865616    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" path="/var/lib/kubelet/pods/767a5bf4-2b1b-494d-bb0f-d9fca4e295d7/volumes"
Nov 02 15:45:27.866362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.866346    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" path="/var/lib/kubelet/pods/92ed44b3-3268-4076-a279-829f1c7facaf/volumes"
Nov 02 15:45:27.866770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.866752    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c120c3-da25-4a39-8a28-916dce067311" path="/var/lib/kubelet/pods/e4c120c3-da25-4a39-8a28-916dce067311/volumes"
Nov 02 15:45:27.919843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.919812    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:27.920236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.920217    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:27.920911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:27.920890    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:45:28.207994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.207919    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"]
Nov 02 15:45:28.208762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208744    7019 cpu_manager.go:414] "containerMap: removing container" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208763    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208774    7019 cpu_manager.go:414] "containerMap: removing container" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208779    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208788    7019 cpu_manager.go:414] "containerMap: removing container" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208792    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208805    7019 cpu_manager.go:414] "containerMap: removing container" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208810    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208816    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208821    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208832    7019 cpu_manager.go:414] "containerMap: removing container" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208837    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208844    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208848    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208856    7019 cpu_manager.go:414] "containerMap: removing container" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208861    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208866    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:28.208864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208871    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208880    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208886    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208893    7019 cpu_manager.go:414] "containerMap: removing container" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208897    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208906    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208910    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208917    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208921    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208929    7019 cpu_manager.go:414] "containerMap: removing container" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208934    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208944    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208949    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208958    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208962    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208969    7019 cpu_manager.go:414] "containerMap: removing container" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208973    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208983    7019 cpu_manager.go:414] "containerMap: removing container" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208988    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208994    7019 cpu_manager.go:414] "containerMap: removing container" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.208999    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209007    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209011    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209022    7019 cpu_manager.go:414] "containerMap: removing container" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209027    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209034    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209039    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209047    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209051    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209060    7019 cpu_manager.go:414] "containerMap: removing container" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209064    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209072    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209077    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209084    7019 cpu_manager.go:414] "containerMap: removing container" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209089    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209105    7019 cpu_manager.go:414] "containerMap: removing container" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209110    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209116    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209120    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209133    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209137    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209144    7019 cpu_manager.go:414] "containerMap: removing container" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209149    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209242    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209249    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209256    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209263    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209268    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209273    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209279    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209285    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209290    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209295    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209300    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209305    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209310    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209316    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209322    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d681fe3-a40e-411a-9a80-8b2c7b90a9d9" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209328    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209334    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209339    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209344    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ed44b3-3268-4076-a279-829f1c7facaf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209350    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209358    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209365    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209370    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b568d3ce-7e0d-4909-8835-f2efc6bd4793" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209375    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209383    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209388    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209394    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209400    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209406    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209413    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209419    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:28.209395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.209424    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:28.214788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.214772    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.233331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.233304    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:45:28.233462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.233406    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:45:28.235293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.235269    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"]
Nov 02 15:45:28.350974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.350937    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.351253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.351230    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.351320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.351288    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdqxj\" (UniqueName: \"kubernetes.io/projected/c674914e-3f08-4722-a12c-1ac77408927a-kube-api-access-mdqxj\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.452351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.452319    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.452519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.452391    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdqxj\" (UniqueName: \"kubernetes.io/projected/c674914e-3f08-4722-a12c-1ac77408927a-kube-api-access-mdqxj\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.452519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.452499    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.453197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.453173    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.454794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.454773    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.480961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.480886    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdqxj\" (UniqueName: \"kubernetes.io/projected/c674914e-3f08-4722-a12c-1ac77408927a-kube-api-access-mdqxj\") pod \"maas-api-revoke-sub-0adb49cc-w9sb9\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.529489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.529451    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:28.850267 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:28.850234    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc674914e_3f08_4722_a12c_1ac77408927a.slice/crio-f118bf4557993806654d39d89c72bec4c689cbf34e45250ed3b20d07491cbdc0 WatchSource:0}: Error finding container f118bf4557993806654d39d89c72bec4c689cbf34e45250ed3b20d07491cbdc0: Status 404 returned error can't find the container with id f118bf4557993806654d39d89c72bec4c689cbf34e45250ed3b20d07491cbdc0
Nov 02 15:45:28.851383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.851355    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"]
Nov 02 15:45:28.959710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:28.958005    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:29.612479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:29.612442    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9" event={"ID":"c674914e-3f08-4722-a12c-1ac77408927a","Type":"ContainerStarted","Data":"89de9ca07b9711c135441dcf79ce5504c930ce7394bfe7a9269ea26a441899a7"}
Nov 02 15:45:29.612699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:29.612504    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9" event={"ID":"c674914e-3f08-4722-a12c-1ac77408927a","Type":"ContainerStarted","Data":"f118bf4557993806654d39d89c72bec4c689cbf34e45250ed3b20d07491cbdc0"}
Nov 02 15:45:29.675549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:29.675485    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9" podStartSLOduration=1.6754659950000002 podStartE2EDuration="1.675465995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:29.672241853 +0000 UTC m=+2923.926195477" watchObservedRunningTime="2026-11-02 15:45:29.675465995 +0000 UTC m=+2923.929419620"
Nov 02 15:45:30.427627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:30.427573    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:30.628919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:30.628871    7019 generic.go:356] "Generic (PLEG): container finished" podID="c674914e-3f08-4722-a12c-1ac77408927a" containerID="89de9ca07b9711c135441dcf79ce5504c930ce7394bfe7a9269ea26a441899a7" exitCode=0
Nov 02 15:45:30.629128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:30.628982    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9" event={"ID":"c674914e-3f08-4722-a12c-1ac77408927a","Type":"ContainerDied","Data":"89de9ca07b9711c135441dcf79ce5504c930ce7394bfe7a9269ea26a441899a7"}
Nov 02 15:45:31.144819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.144780    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"]
Nov 02 15:45:31.145495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145479    7019 cpu_manager.go:414] "containerMap: removing container" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145497    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98247b5f-4631-4cbe-9fc7-a447913966b7" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145511    7019 cpu_manager.go:414] "containerMap: removing container" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145517    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145527    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145533    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145547    7019 cpu_manager.go:414] "containerMap: removing container" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145552    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:31.145560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145561    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145566    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2915eaf5-dbe8-4df0-b6c9-ab65dc888e8b" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145584    7019 cpu_manager.go:414] "containerMap: removing container" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145589    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145598    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145603    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145617    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145622    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c120c3-da25-4a39-8a28-916dce067311" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145658    7019 cpu_manager.go:414] "containerMap: removing container" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145663    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145775    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7b87a4-682a-4462-9610-63ee828259e3" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145785    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="75fa808b-30f1-4090-859a-c8bb8fd9a999" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145793    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="767a5bf4-2b1b-494d-bb0f-d9fca4e295d7" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145798    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a625702-a861-4efb-9bf5-b50c4f3119cf" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145807    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="08cc86fb-8725-48e1-9a73-7ad63bc05755" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145815    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="935acbcf-5b16-4096-8a48-73579d2b6505" containerName="cleanup"
Nov 02 15:45:31.145834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.145827    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bca953-9136-42bd-916d-1e3f1915b53a" containerName="cleanup"
Nov 02 15:45:31.149435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.149418    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.173835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.173805    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"]
Nov 02 15:45:31.183909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.183877    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-fda646-openshift-default-dockercfg-kdct4\""
Nov 02 15:45:31.206482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.206437    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.206703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.206572    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.206703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.206622    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.206703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.206673    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589f2b68-0b46-4320-880d-ff9e9d047d18-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.206921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.206769    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.206921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.206803    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.207041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.206986    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.207041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.207021    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb975\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-kube-api-access-hb975\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.207156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.207099    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.308636 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.308599    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.308850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.308694    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589f2b68-0b46-4320-880d-ff9e9d047d18-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.308850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.308761    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.308850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.308798    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.308940    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.308971    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb975\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-kube-api-access-hb975\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309016    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309071    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-credential-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309214    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309283    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-socket\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309323    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-certs\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309325    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309428    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589f2b68-0b46-4320-880d-ff9e9d047d18-istiod-ca-cert\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.309527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.309508    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-data\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.311265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.311232    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-envoy\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.311561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.311543    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-podinfo\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.335150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.335126    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-token\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.335295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.335267    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb975\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-kube-api-access-hb975\") pod \"e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") " pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.464924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.464841    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:31.795841 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:31.795809    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589f2b68_0b46_4320_880d_ff9e9d047d18.slice/crio-59d643f5ee3d5cf32627cd390aee0e469f7d4836162e4e0dd830e765d4054518 WatchSource:0}: Error finding container 59d643f5ee3d5cf32627cd390aee0e469f7d4836162e4e0dd830e765d4054518: Status 404 returned error can't find the container with id 59d643f5ee3d5cf32627cd390aee0e469f7d4836162e4e0dd830e765d4054518
Nov 02 15:45:31.797602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.797571    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"]
Nov 02 15:45:31.798982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.798945    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:31.799071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.799026    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:31.799071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.799061    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:45:31.807065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.807032    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:31.922342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.922303    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-xks-ca\") pod \"c674914e-3f08-4722-a12c-1ac77408927a\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") "
Nov 02 15:45:31.922603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.922538    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c674914e-3f08-4722-a12c-1ac77408927a-kube-api-access-mdqxj\" (UniqueName: \"kubernetes.io/projected/c674914e-3f08-4722-a12c-1ac77408927a-kube-api-access-mdqxj\") pod \"c674914e-3f08-4722-a12c-1ac77408927a\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") "
Nov 02 15:45:31.922603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.922582    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-openshift-ca\") pod \"c674914e-3f08-4722-a12c-1ac77408927a\" (UID: \"c674914e-3f08-4722-a12c-1ac77408927a\") "
Nov 02 15:45:31.924916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.924888    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-openshift-ca" pod "c674914e-3f08-4722-a12c-1ac77408927a" (UID: "c674914e-3f08-4722-a12c-1ac77408927a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:31.927490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.927462    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-xks-ca" pod "c674914e-3f08-4722-a12c-1ac77408927a" (UID: "c674914e-3f08-4722-a12c-1ac77408927a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:31.933868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:31.933843    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c674914e-3f08-4722-a12c-1ac77408927a-kube-api-access-mdqxj" pod "c674914e-3f08-4722-a12c-1ac77408927a" (UID: "c674914e-3f08-4722-a12c-1ac77408927a"). InnerVolumeSpecName "kube-api-access-mdqxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:32.026896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.026862    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdqxj\" (UniqueName: \"kubernetes.io/projected/c674914e-3f08-4722-a12c-1ac77408927a-kube-api-access-mdqxj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:32.026896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.026901    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:32.027143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.026911    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c674914e-3f08-4722-a12c-1ac77408927a-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:32.041297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.041203    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:32.634930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.634889    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"]
Nov 02 15:45:32.635828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.635805    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c674914e-3f08-4722-a12c-1ac77408927a" containerName="revoke-keys"
Nov 02 15:45:32.635828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.635828    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c674914e-3f08-4722-a12c-1ac77408927a" containerName="revoke-keys"
Nov 02 15:45:32.636072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.636057    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c674914e-3f08-4722-a12c-1ac77408927a" containerName="revoke-keys"
Nov 02 15:45:32.640290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.640267    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.655226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.655193    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9" event={"ID":"c674914e-3f08-4722-a12c-1ac77408927a","Type":"ContainerDied","Data":"f118bf4557993806654d39d89c72bec4c689cbf34e45250ed3b20d07491cbdc0"}
Nov 02 15:45:32.655329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.655253    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f118bf4557993806654d39d89c72bec4c689cbf34e45250ed3b20d07491cbdc0"
Nov 02 15:45:32.655329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.655320    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"
Nov 02 15:45:32.661101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.660614    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"]
Nov 02 15:45:32.713577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.713518    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" event={"ID":"589f2b68-0b46-4320-880d-ff9e9d047d18","Type":"ContainerStarted","Data":"50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473"}
Nov 02 15:45:32.714018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.713976    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" event={"ID":"589f2b68-0b46-4320-880d-ff9e9d047d18","Type":"ContainerStarted","Data":"59d643f5ee3d5cf32627cd390aee0e469f7d4836162e4e0dd830e765d4054518"}
Nov 02 15:45:32.741068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.741036    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmx5q\" (UniqueName: \"kubernetes.io/projected/78062598-34cb-4f28-a80e-a669c553b28f-kube-api-access-tmx5q\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.741315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.741290    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.741775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.741760    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.805567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.805491    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podStartSLOduration=1.805471495 podStartE2EDuration="1.805471495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:32.798851589 +0000 UTC m=+2927.052805214" watchObservedRunningTime="2026-11-02 15:45:32.805471495 +0000 UTC m=+2927.059425121"
Nov 02 15:45:32.842967 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.842933    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.843162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.843022    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.843162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.843124    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmx5q\" (UniqueName: \"kubernetes.io/projected/78062598-34cb-4f28-a80e-a669c553b28f-kube-api-access-tmx5q\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.843716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.843689    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.845314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.845293    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:32.894722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:32.893711    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmx5q\" (UniqueName: \"kubernetes.io/projected/78062598-34cb-4f28-a80e-a669c553b28f-kube-api-access-tmx5q\") pod \"maas-api-revoke-sub-e57edf19-bw7vn\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:33.013952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.013911    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:33.244004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.243924    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:33.447699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.442251    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:33.513721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.493777    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:33.530856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.530412    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 02 15:45:33.530856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.530492    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 02 15:45:33.532744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.532658    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"]
Nov 02 15:45:33.533201 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:33.533168    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78062598_34cb_4f28_a80e_a669c553b28f.slice/crio-e6bc8386716b0c97e38e1568b11ebe8004e0ad237549cedb97e6f74cd0265d1a WatchSource:0}: Error finding container e6bc8386716b0c97e38e1568b11ebe8004e0ad237549cedb97e6f74cd0265d1a: Status 404 returned error can't find the container with id e6bc8386716b0c97e38e1568b11ebe8004e0ad237549cedb97e6f74cd0265d1a
Nov 02 15:45:33.835733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:33.832092    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn" event={"ID":"78062598-34cb-4f28-a80e-a669c553b28f","Type":"ContainerStarted","Data":"e6bc8386716b0c97e38e1568b11ebe8004e0ad237549cedb97e6f74cd0265d1a"}
Nov 02 15:45:34.401629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:34.401598    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:34.482885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:34.476505    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 02 15:45:34.482885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:34.476573    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 02 15:45:34.841498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:34.841461    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn" event={"ID":"78062598-34cb-4f28-a80e-a669c553b28f","Type":"ContainerStarted","Data":"d7f2ecb220e5b0131de2dc4e5a48cf9c88b73dac319c985d8030d4d2bb4e39f6"}
Nov 02 15:45:34.902805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:34.902739    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn" podStartSLOduration=2.902720444 podStartE2EDuration="2.902720444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:34.901432638 +0000 UTC m=+2929.155386299" watchObservedRunningTime="2026-11-02 15:45:34.902720444 +0000 UTC m=+2929.156674068"
Nov 02 15:45:35.477670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:35.477053    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 02 15:45:35.477670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:35.477136    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 02 15:45:35.852088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:35.851979    7019 generic.go:356] "Generic (PLEG): container finished" podID="78062598-34cb-4f28-a80e-a669c553b28f" containerID="d7f2ecb220e5b0131de2dc4e5a48cf9c88b73dac319c985d8030d4d2bb4e39f6" exitCode=0
Nov 02 15:45:35.852696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:35.852073    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn" event={"ID":"78062598-34cb-4f28-a80e-a669c553b28f","Type":"ContainerDied","Data":"d7f2ecb220e5b0131de2dc4e5a48cf9c88b73dac319c985d8030d4d2bb4e39f6"}
Nov 02 15:45:35.954398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:35.954360    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:36.468673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:36.466040    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 02 15:45:36.468673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:36.466119    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 02 15:45:37.130802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.130774    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:37.259175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.259141    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-openshift-ca\") pod \"78062598-34cb-4f28-a80e-a669c553b28f\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") "
Nov 02 15:45:37.259348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.259189    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-xks-ca\") pod \"78062598-34cb-4f28-a80e-a669c553b28f\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") "
Nov 02 15:45:37.259440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.259400    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78062598-34cb-4f28-a80e-a669c553b28f-kube-api-access-tmx5q\" (UniqueName: \"kubernetes.io/projected/78062598-34cb-4f28-a80e-a669c553b28f-kube-api-access-tmx5q\") pod \"78062598-34cb-4f28-a80e-a669c553b28f\" (UID: \"78062598-34cb-4f28-a80e-a669c553b28f\") "
Nov 02 15:45:37.259553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.259531    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-openshift-ca" pod "78062598-34cb-4f28-a80e-a669c553b28f" (UID: "78062598-34cb-4f28-a80e-a669c553b28f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:37.260283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.260246    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:37.261396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.261373    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-xks-ca" pod "78062598-34cb-4f28-a80e-a669c553b28f" (UID: "78062598-34cb-4f28-a80e-a669c553b28f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:37.261508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.261463    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78062598-34cb-4f28-a80e-a669c553b28f-kube-api-access-tmx5q" pod "78062598-34cb-4f28-a80e-a669c553b28f" (UID: "78062598-34cb-4f28-a80e-a669c553b28f"). InnerVolumeSpecName "kube-api-access-tmx5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:37.361656 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.361622    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmx5q\" (UniqueName: \"kubernetes.io/projected/78062598-34cb-4f28-a80e-a669c553b28f-kube-api-access-tmx5q\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:37.361813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.361687    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78062598-34cb-4f28-a80e-a669c553b28f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:37.469731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.469074    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 02 15:45:37.469731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.469136    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 02 15:45:37.603230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.603197    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:37.917122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.917092    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"
Nov 02 15:45:37.917922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.917130    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn" event={"ID":"78062598-34cb-4f28-a80e-a669c553b28f","Type":"ContainerDied","Data":"e6bc8386716b0c97e38e1568b11ebe8004e0ad237549cedb97e6f74cd0265d1a"}
Nov 02 15:45:37.918050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:37.917976    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6bc8386716b0c97e38e1568b11ebe8004e0ad237549cedb97e6f74cd0265d1a"
Nov 02 15:45:38.465611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.465222    7019 patch_prober.go:29] interesting pod/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 02 15:45:38.465611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.465286    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 02 15:45:38.723961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.723879    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"]
Nov 02 15:45:38.724632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.724617    7019 cpu_manager.go:414] "containerMap: removing container" podUID="78062598-34cb-4f28-a80e-a669c553b28f" containerName="revoke-keys"
Nov 02 15:45:38.724707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.724635    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78062598-34cb-4f28-a80e-a669c553b28f" containerName="revoke-keys"
Nov 02 15:45:38.724790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.724780    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="78062598-34cb-4f28-a80e-a669c553b28f" containerName="revoke-keys"
Nov 02 15:45:38.728746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.728726    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:38.757534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.757496    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"]
Nov 02 15:45:38.759935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.759911    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:45:38.760223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.760206    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:45:38.899504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.899465    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:38.899763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.899741    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2sd\" (UniqueName: \"kubernetes.io/projected/912f5584-3ecc-49ce-809a-eb836e0d08a2-kube-api-access-2z2sd\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:38.899968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:38.899940    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.001830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.001738    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.001830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.001802    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.002085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.002029    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2sd\" (UniqueName: \"kubernetes.io/projected/912f5584-3ecc-49ce-809a-eb836e0d08a2-kube-api-access-2z2sd\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.002565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.002535    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.004385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.004351    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.028516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.028487    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2sd\" (UniqueName: \"kubernetes.io/projected/912f5584-3ecc-49ce-809a-eb836e0d08a2-kube-api-access-2z2sd\") pod \"maas-api-revoke-sub-df27c9e1-5h8ft\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.056515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.056481    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:39.256686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.256584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:39.371080 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:39.371049    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912f5584_3ecc_49ce_809a_eb836e0d08a2.slice/crio-8b414641ba5c1732a5d5aaf9eb5f61cacee685481dae17ba4076e65a24175eb1 WatchSource:0}: Error finding container 8b414641ba5c1732a5d5aaf9eb5f61cacee685481dae17ba4076e65a24175eb1: Status 404 returned error can't find the container with id 8b414641ba5c1732a5d5aaf9eb5f61cacee685481dae17ba4076e65a24175eb1
Nov 02 15:45:39.371933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.371905    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"]
Nov 02 15:45:39.471877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.471850    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:39.472329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.472315    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:39.473217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.473196    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:45:39.936725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.936686    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft" event={"ID":"912f5584-3ecc-49ce-809a-eb836e0d08a2","Type":"ContainerStarted","Data":"b8a68f2c989bd77946cb7c3c2c5782afbf438a04506d4b0a00b235b4eac136f6"}
Nov 02 15:45:39.937083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.936766    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft" event={"ID":"912f5584-3ecc-49ce-809a-eb836e0d08a2","Type":"ContainerStarted","Data":"8b414641ba5c1732a5d5aaf9eb5f61cacee685481dae17ba4076e65a24175eb1"}
Nov 02 15:45:39.985218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:39.985167    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft" podStartSLOduration=1.985153604 podStartE2EDuration="1.985153604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:39.981963212 +0000 UTC m=+2934.235916835" watchObservedRunningTime="2026-11-02 15:45:39.985153604 +0000 UTC m=+2934.239107224"
Nov 02 15:45:40.745661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:40.745598    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:40.920761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:40.917151    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:40.950894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:40.950862    7019 generic.go:356] "Generic (PLEG): container finished" podID="912f5584-3ecc-49ce-809a-eb836e0d08a2" containerID="b8a68f2c989bd77946cb7c3c2c5782afbf438a04506d4b0a00b235b4eac136f6" exitCode=0
Nov 02 15:45:40.951077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:40.950907    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft" event={"ID":"912f5584-3ecc-49ce-809a-eb836e0d08a2","Type":"ContainerDied","Data":"b8a68f2c989bd77946cb7c3c2c5782afbf438a04506d4b0a00b235b4eac136f6"}
Nov 02 15:45:41.064455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:41.064037    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:41.314730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:41.296241    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:41.421670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:41.420794    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"]
Nov 02 15:45:41.446671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:41.427413    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9cfd1f-qrzpl"]
Nov 02 15:45:41.461545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:41.448563    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:41.866197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:41.866160    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5437a45-014a-473a-9178-227e9a494372" path="/var/lib/kubelet/pods/a5437a45-014a-473a-9178-227e9a494372/volumes"
Nov 02 15:45:41.967215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:41.967184    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:42.130792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.130736    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:42.222429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.222387    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-openshift-ca\") pod \"912f5584-3ecc-49ce-809a-eb836e0d08a2\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") "
Nov 02 15:45:42.222602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.222521    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912f5584-3ecc-49ce-809a-eb836e0d08a2-kube-api-access-2z2sd\" (UniqueName: \"kubernetes.io/projected/912f5584-3ecc-49ce-809a-eb836e0d08a2-kube-api-access-2z2sd\") pod \"912f5584-3ecc-49ce-809a-eb836e0d08a2\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") "
Nov 02 15:45:42.222730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.222635    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-xks-ca\") pod \"912f5584-3ecc-49ce-809a-eb836e0d08a2\" (UID: \"912f5584-3ecc-49ce-809a-eb836e0d08a2\") "
Nov 02 15:45:42.223104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.223072    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-openshift-ca" pod "912f5584-3ecc-49ce-809a-eb836e0d08a2" (UID: "912f5584-3ecc-49ce-809a-eb836e0d08a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:42.223505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.223466    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:42.227355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.225600    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-xks-ca" pod "912f5584-3ecc-49ce-809a-eb836e0d08a2" (UID: "912f5584-3ecc-49ce-809a-eb836e0d08a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:42.227355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.226192    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912f5584-3ecc-49ce-809a-eb836e0d08a2-kube-api-access-2z2sd" pod "912f5584-3ecc-49ce-809a-eb836e0d08a2" (UID: "912f5584-3ecc-49ce-809a-eb836e0d08a2"). InnerVolumeSpecName "kube-api-access-2z2sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:42.324713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.324641    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z2sd\" (UniqueName: \"kubernetes.io/projected/912f5584-3ecc-49ce-809a-eb836e0d08a2-kube-api-access-2z2sd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:42.324713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.324716    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912f5584-3ecc-49ce-809a-eb836e0d08a2-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:42.971136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.971106    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"
Nov 02 15:45:42.971572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.971148    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft" event={"ID":"912f5584-3ecc-49ce-809a-eb836e0d08a2","Type":"ContainerDied","Data":"8b414641ba5c1732a5d5aaf9eb5f61cacee685481dae17ba4076e65a24175eb1"}
Nov 02 15:45:42.971572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:42.971207    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b414641ba5c1732a5d5aaf9eb5f61cacee685481dae17ba4076e65a24175eb1"
Nov 02 15:45:43.104440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.104408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:43.467696 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.467664    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"]
Nov 02 15:45:43.468416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.468395    7019 cpu_manager.go:414] "containerMap: removing container" podUID="912f5584-3ecc-49ce-809a-eb836e0d08a2" containerName="revoke-keys"
Nov 02 15:45:43.468545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.468417    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912f5584-3ecc-49ce-809a-eb836e0d08a2" containerName="revoke-keys"
Nov 02 15:45:43.468697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.468677    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="912f5584-3ecc-49ce-809a-eb836e0d08a2" containerName="revoke-keys"
Nov 02 15:45:43.472591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.472561    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.492683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.492621    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"]
Nov 02 15:45:43.545766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.545734    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg45r\" (UniqueName: \"kubernetes.io/projected/525a417a-3500-453e-ab80-92aea18e3468-kube-api-access-dg45r\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.545926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.545881    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a417a-3500-453e-ab80-92aea18e3468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.545984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.545965    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a417a-3500-453e-ab80-92aea18e3468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.572169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.572140    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:45:43.572361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.572348    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:45:43.647887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.647859    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg45r\" (UniqueName: \"kubernetes.io/projected/525a417a-3500-453e-ab80-92aea18e3468-kube-api-access-dg45r\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.648073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.647939    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a417a-3500-453e-ab80-92aea18e3468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.648073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.647970    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a417a-3500-453e-ab80-92aea18e3468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.648552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.648532    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a417a-3500-453e-ab80-92aea18e3468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.650300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.650275    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a417a-3500-453e-ab80-92aea18e3468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.674187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.674159    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg45r\" (UniqueName: \"kubernetes.io/projected/525a417a-3500-453e-ab80-92aea18e3468-kube-api-access-dg45r\") pod \"maas-api-revoke-sub-2d776d7b-nh2nr\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:43.790729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:43.790601    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:44.103928 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:44.103897    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525a417a_3500_453e_ab80_92aea18e3468.slice/crio-ac535c5d274c04eb43994799298289d230118e3970c9d1b12f1636b71faaf6cf WatchSource:0}: Error finding container ac535c5d274c04eb43994799298289d230118e3970c9d1b12f1636b71faaf6cf: Status 404 returned error can't find the container with id ac535c5d274c04eb43994799298289d230118e3970c9d1b12f1636b71faaf6cf
Nov 02 15:45:44.105661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:44.105629    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"]
Nov 02 15:45:44.761146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:44.761075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:44.995256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:44.995199    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr" event={"ID":"525a417a-3500-453e-ab80-92aea18e3468","Type":"ContainerStarted","Data":"644864ccddfea93c90b2c509beb1a5c3629899441d64d53ec48bd9cac83bbebe"}
Nov 02 15:45:44.995256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:44.995263    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr" event={"ID":"525a417a-3500-453e-ab80-92aea18e3468","Type":"ContainerStarted","Data":"ac535c5d274c04eb43994799298289d230118e3970c9d1b12f1636b71faaf6cf"}
Nov 02 15:45:45.045271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:45.045173    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr" podStartSLOduration=2.045158604 podStartE2EDuration="2.045158604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:45.041132471 +0000 UTC m=+2939.295086095" watchObservedRunningTime="2026-11-02 15:45:45.045158604 +0000 UTC m=+2939.299112225"
Nov 02 15:45:46.008511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:46.008477    7019 generic.go:356] "Generic (PLEG): container finished" podID="525a417a-3500-453e-ab80-92aea18e3468" containerID="644864ccddfea93c90b2c509beb1a5c3629899441d64d53ec48bd9cac83bbebe" exitCode=0
Nov 02 15:45:46.008992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:46.008575    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr" event={"ID":"525a417a-3500-453e-ab80-92aea18e3468","Type":"ContainerDied","Data":"644864ccddfea93c90b2c509beb1a5c3629899441d64d53ec48bd9cac83bbebe"}
Nov 02 15:45:46.474197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:46.474124    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:47.190198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.190172    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:47.310713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.310617    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525a417a-3500-453e-ab80-92aea18e3468-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a417a-3500-453e-ab80-92aea18e3468-maas-api-xks-ca\") pod \"525a417a-3500-453e-ab80-92aea18e3468\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") "
Nov 02 15:45:47.310866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.310771    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525a417a-3500-453e-ab80-92aea18e3468-kube-api-access-dg45r\" (UniqueName: \"kubernetes.io/projected/525a417a-3500-453e-ab80-92aea18e3468-kube-api-access-dg45r\") pod \"525a417a-3500-453e-ab80-92aea18e3468\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") "
Nov 02 15:45:47.310958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.310935    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525a417a-3500-453e-ab80-92aea18e3468-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a417a-3500-453e-ab80-92aea18e3468-maas-api-openshift-ca\") pod \"525a417a-3500-453e-ab80-92aea18e3468\" (UID: \"525a417a-3500-453e-ab80-92aea18e3468\") "
Nov 02 15:45:47.311346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.311316    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525a417a-3500-453e-ab80-92aea18e3468-maas-api-openshift-ca" pod "525a417a-3500-453e-ab80-92aea18e3468" (UID: "525a417a-3500-453e-ab80-92aea18e3468"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:47.311948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.311927    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525a417a-3500-453e-ab80-92aea18e3468-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:47.313008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.312986    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525a417a-3500-453e-ab80-92aea18e3468-maas-api-xks-ca" pod "525a417a-3500-453e-ab80-92aea18e3468" (UID: "525a417a-3500-453e-ab80-92aea18e3468"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:47.313143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.313121    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525a417a-3500-453e-ab80-92aea18e3468-kube-api-access-dg45r" pod "525a417a-3500-453e-ab80-92aea18e3468" (UID: "525a417a-3500-453e-ab80-92aea18e3468"). InnerVolumeSpecName "kube-api-access-dg45r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:47.413075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.413042    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525a417a-3500-453e-ab80-92aea18e3468-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:47.413075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.413069    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg45r\" (UniqueName: \"kubernetes.io/projected/525a417a-3500-453e-ab80-92aea18e3468-kube-api-access-dg45r\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:47.972511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:47.972469    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:48.051993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:48.051955    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr" event={"ID":"525a417a-3500-453e-ab80-92aea18e3468","Type":"ContainerDied","Data":"ac535c5d274c04eb43994799298289d230118e3970c9d1b12f1636b71faaf6cf"}
Nov 02 15:45:48.052184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:48.052053    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac535c5d274c04eb43994799298289d230118e3970c9d1b12f1636b71faaf6cf"
Nov 02 15:45:48.052184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:48.052121    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"
Nov 02 15:45:48.305811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:48.305718    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:48.460786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:48.460751    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:48.681786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:48.681746    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:49.181518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.181479    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"]
Nov 02 15:45:49.182322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.182299    7019 cpu_manager.go:414] "containerMap: removing container" podUID="525a417a-3500-453e-ab80-92aea18e3468" containerName="revoke-keys"
Nov 02 15:45:49.182322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.182320    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525a417a-3500-453e-ab80-92aea18e3468" containerName="revoke-keys"
Nov 02 15:45:49.182497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.182484    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="525a417a-3500-453e-ab80-92aea18e3468" containerName="revoke-keys"
Nov 02 15:45:49.186292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.186274    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.207214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.207185    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"]
Nov 02 15:45:49.274342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.274306    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:45:49.274502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.274468    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:45:49.291462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.291427    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.291621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.291468    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5f8t\" (UniqueName: \"kubernetes.io/projected/f723fed1-51c5-49a6-83a2-f8bf62ddef89-kube-api-access-s5f8t\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.291801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.291772    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.393688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.393404    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.395043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.394235    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.395043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.394291    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5f8t\" (UniqueName: \"kubernetes.io/projected/f723fed1-51c5-49a6-83a2-f8bf62ddef89-kube-api-access-s5f8t\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.395043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.394641    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.397701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.397681    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.421395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.421365    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5f8t\" (UniqueName: \"kubernetes.io/projected/f723fed1-51c5-49a6-83a2-f8bf62ddef89-kube-api-access-s5f8t\") pod \"maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.502143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.502060    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:49.641728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.641602    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:49.674075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.673637    7019 scope.go:122] "RemoveContainer" containerID="4a75748ba2c4e64443861cd0fb888a2e5e9f53652506813ef04880307bd3bdc5"
Nov 02 15:45:49.686212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.686174    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"]
Nov 02 15:45:49.876677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:49.873708    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:50.028938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:50.027977    7019 scope.go:122] "RemoveContainer" containerID="b6fd824c9ce202a296f0274d4940bbb70cebfdd967396b046b8aae0535c9af3f"
Nov 02 15:45:50.028938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:50.028491    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:50.032792 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:50.029730    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf723fed1_51c5_49a6_83a2_f8bf62ddef89.slice/crio-f78e1551de1bff36ccd53ab01890597f2bb97795453a82f940a852fe538594d3 WatchSource:0}: Error finding container f78e1551de1bff36ccd53ab01890597f2bb97795453a82f940a852fe538594d3: Status 404 returned error can't find the container with id f78e1551de1bff36ccd53ab01890597f2bb97795453a82f940a852fe538594d3
Nov 02 15:45:50.036142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:50.036117    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"]
Nov 02 15:45:50.089563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:50.089523    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2" event={"ID":"f723fed1-51c5-49a6-83a2-f8bf62ddef89","Type":"ContainerStarted","Data":"f78e1551de1bff36ccd53ab01890597f2bb97795453a82f940a852fe538594d3"}
Nov 02 15:45:50.250764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:50.250732    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:50.478502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:50.478468    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:50.745733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:50.736817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:51.102075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:51.101974    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2" event={"ID":"f723fed1-51c5-49a6-83a2-f8bf62ddef89","Type":"ContainerStarted","Data":"b4c8aa292aea7f44c1c3a8adcba341332cc13c26f225e82cfdd5d7b57087c2da"}
Nov 02 15:45:51.150493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:51.150431    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2" podStartSLOduration=2.150418503 podStartE2EDuration="2.150418503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:51.147008615 +0000 UTC m=+2945.400962239" watchObservedRunningTime="2026-11-02 15:45:51.150418503 +0000 UTC m=+2945.404372137"
Nov 02 15:45:51.407285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:51.407244    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:52.115058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:52.115017    7019 generic.go:356] "Generic (PLEG): container finished" podID="f723fed1-51c5-49a6-83a2-f8bf62ddef89" containerID="b4c8aa292aea7f44c1c3a8adcba341332cc13c26f225e82cfdd5d7b57087c2da" exitCode=0
Nov 02 15:45:52.115528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:52.115078    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2" event={"ID":"f723fed1-51c5-49a6-83a2-f8bf62ddef89","Type":"ContainerDied","Data":"b4c8aa292aea7f44c1c3a8adcba341332cc13c26f225e82cfdd5d7b57087c2da"}
Nov 02 15:45:52.713090 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:45:52.713051    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:45:53.042846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.042761    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:53.315200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.315129    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:53.392142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.392106    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f723fed1-51c5-49a6-83a2-f8bf62ddef89-kube-api-access-s5f8t\" (UniqueName: \"kubernetes.io/projected/f723fed1-51c5-49a6-83a2-f8bf62ddef89-kube-api-access-s5f8t\") pod \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") "
Nov 02 15:45:53.392336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.392181    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-xks-ca\") pod \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") "
Nov 02 15:45:53.392336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.392253    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-openshift-ca\") pod \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\" (UID: \"f723fed1-51c5-49a6-83a2-f8bf62ddef89\") "
Nov 02 15:45:53.392773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.392740    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-openshift-ca" pod "f723fed1-51c5-49a6-83a2-f8bf62ddef89" (UID: "f723fed1-51c5-49a6-83a2-f8bf62ddef89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:53.393196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.393168    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:53.395708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.394885    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-xks-ca" pod "f723fed1-51c5-49a6-83a2-f8bf62ddef89" (UID: "f723fed1-51c5-49a6-83a2-f8bf62ddef89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:53.397843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.397816    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f723fed1-51c5-49a6-83a2-f8bf62ddef89-kube-api-access-s5f8t" pod "f723fed1-51c5-49a6-83a2-f8bf62ddef89" (UID: "f723fed1-51c5-49a6-83a2-f8bf62ddef89"). InnerVolumeSpecName "kube-api-access-s5f8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:53.494897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.494861    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5f8t\" (UniqueName: \"kubernetes.io/projected/f723fed1-51c5-49a6-83a2-f8bf62ddef89-kube-api-access-s5f8t\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:53.494897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:53.494891    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f723fed1-51c5-49a6-83a2-f8bf62ddef89-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:54.038959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.038916    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:54.083519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.083486    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"]
Nov 02 15:45:54.084641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.084620    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f723fed1-51c5-49a6-83a2-f8bf62ddef89" containerName="revoke-keys"
Nov 02 15:45:54.084939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.084915    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f723fed1-51c5-49a6-83a2-f8bf62ddef89" containerName="revoke-keys"
Nov 02 15:45:54.085141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.085126    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f723fed1-51c5-49a6-83a2-f8bf62ddef89" containerName="revoke-keys"
Nov 02 15:45:54.089799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.089779    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.106778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.106752    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"]
Nov 02 15:45:54.134065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.133977    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2" event={"ID":"f723fed1-51c5-49a6-83a2-f8bf62ddef89","Type":"ContainerDied","Data":"f78e1551de1bff36ccd53ab01890597f2bb97795453a82f940a852fe538594d3"}
Nov 02 15:45:54.134065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.134025    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f78e1551de1bff36ccd53ab01890597f2bb97795453a82f940a852fe538594d3"
Nov 02 15:45:54.134282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.134078    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"
Nov 02 15:45:54.187813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.187767    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:54.213147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.213107    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrc7\" (UniqueName: \"kubernetes.io/projected/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-kube-api-access-ldrc7\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.213395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.213365    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.213518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.213499    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.314855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.314814    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.315035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.314996    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrc7\" (UniqueName: \"kubernetes.io/projected/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-kube-api-access-ldrc7\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.315136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.315123    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.316022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.315986    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.317483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.317462    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.340807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.340782    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrc7\" (UniqueName: \"kubernetes.io/projected/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-kube-api-access-ldrc7\") pod \"maas-api-revoke-sub-99a4c14d-z6zf6\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.410196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.410112    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:54.717310 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:45:54.717281    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0b45ea_4edb_42e6_95e3_cfb157c0ba3e.slice/crio-4d332c30117663189e8f2d560bf743998fdc695325e43a187816a51dd4c841a8 WatchSource:0}: Error finding container 4d332c30117663189e8f2d560bf743998fdc695325e43a187816a51dd4c841a8: Status 404 returned error can't find the container with id 4d332c30117663189e8f2d560bf743998fdc695325e43a187816a51dd4c841a8
Nov 02 15:45:54.719049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.719025    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"]
Nov 02 15:45:54.883686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:54.883629    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:55.143195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:55.143153    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6" event={"ID":"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e","Type":"ContainerStarted","Data":"4d332c30117663189e8f2d560bf743998fdc695325e43a187816a51dd4c841a8"}
Nov 02 15:45:55.748848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:55.748679    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:55.892028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:55.891993    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:56.280128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:56.279418    7019 generic.go:356] "Generic (PLEG): container finished" podID="ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" containerID="f3f0d43afa26611d53fcfd424804754daf6f0c36be20d369e5bd7e2e15a7ea36" exitCode=0
Nov 02 15:45:56.280128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:56.279727    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6" event={"ID":"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e","Type":"ContainerDied","Data":"f3f0d43afa26611d53fcfd424804754daf6f0c36be20d369e5bd7e2e15a7ea36"}
Nov 02 15:45:56.698766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:56.698731    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:57.500341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.500307    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:57.576836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.576747    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-openshift-ca\") pod \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") "
Nov 02 15:45:57.577034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.576921    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-kube-api-access-ldrc7\" (UniqueName: \"kubernetes.io/projected/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-kube-api-access-ldrc7\") pod \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") "
Nov 02 15:45:57.577190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.577161    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-openshift-ca" pod "ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" (UID: "ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:45:57.577618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.577590    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-xks-ca\") pod \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\" (UID: \"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e\") "
Nov 02 15:45:57.578341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.578310    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:57.579826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.579797    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-kube-api-access-ldrc7" pod "ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" (UID: "ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e"). InnerVolumeSpecName "kube-api-access-ldrc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:45:57.580502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.580460    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-xks-ca" pod "ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" (UID: "ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:45:57.679875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.679833    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:57.679875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:57.679865    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrc7\" (UniqueName: \"kubernetes.io/projected/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e-kube-api-access-ldrc7\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:45:58.098691 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.098616    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:58.300938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.300846    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:58.304878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.304842    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6" event={"ID":"ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e","Type":"ContainerDied","Data":"4d332c30117663189e8f2d560bf743998fdc695325e43a187816a51dd4c841a8"}
Nov 02 15:45:58.305027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.304902    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d332c30117663189e8f2d560bf743998fdc695325e43a187816a51dd4c841a8"
Nov 02 15:45:58.305027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.304968    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"
Nov 02 15:45:58.475265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.475231    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:58.735688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.730592    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:58.915958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:58.909928    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:45:59.367776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:45:59.367722    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:00.737265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:00.737216    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:02.586324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:02.586218    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:02.768819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:02.757460    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:04.374724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:04.374685    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:04.525603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:04.525549    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:04.673319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:04.673174    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:04.878619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:04.878580    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:04.886140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:04.886104    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" podUID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" containerName="main" containerID="cri-o://047da94fe54fd066472ac19cecfb794614adb064c24c6326e6ed7a9906422db9" gracePeriod=45
Nov 02 15:46:05.158262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.158103    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"]
Nov 02 15:46:05.158262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.158166    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3805b8-8tsn2"]
Nov 02 15:46:05.203102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.203064    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" podUID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.215:8000/ready\": EOF"
Nov 02 15:46:05.204898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.204866    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:05.394734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.394694    7019 generic.go:356] "Generic (PLEG): container finished" podID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" containerID="047da94fe54fd066472ac19cecfb794614adb064c24c6326e6ed7a9906422db9" exitCode=0
Nov 02 15:46:05.395143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.394744    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" event={"ID":"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4","Type":"ContainerDied","Data":"047da94fe54fd066472ac19cecfb794614adb064c24c6326e6ed7a9906422db9"}
Nov 02 15:46:05.451185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.451146    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:05.872781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.872662    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f723fed1-51c5-49a6-83a2-f8bf62ddef89" path="/var/lib/kubelet/pods/f723fed1-51c5-49a6-83a2-f8bf62ddef89/volumes"
Nov 02 15:46:05.912097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.912068    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:46:05.949824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.949791    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tls-certs\" (UniqueName: \"kubernetes.io/secret/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tls-certs\") pod \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") "
Nov 02 15:46:05.950213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.950187    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-model-cache\") pod \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") "
Nov 02 15:46:05.950439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.950418    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-kube-api-access-vvrnc\" (UniqueName: \"kubernetes.io/projected/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-kube-api-access-vvrnc\") pod \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") "
Nov 02 15:46:05.950609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.950594    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-home\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-home\") pod \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") "
Nov 02 15:46:05.950911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.950886    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-model-cache" pod "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" (UID: "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:05.951025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.950626    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:05.951522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.951502    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-home" pod "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" (UID: "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:05.951777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.951759    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tmp-dir\") pod \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") "
Nov 02 15:46:05.951924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.951910    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-dshm\") pod \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\" (UID: \"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4\") "
Nov 02 15:46:05.952836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.952815    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:05.953079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.953036    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:05.953755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:05.953720    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tmp-dir" pod "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" (UID: "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:06.046756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.046699    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tls-certs" pod "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" (UID: "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:06.069713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.069584    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:06.102683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.102090    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-kube-api-access-vvrnc" pod "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" (UID: "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4"). InnerVolumeSpecName "kube-api-access-vvrnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:06.131682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.121464    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-dshm" pod "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" (UID: "03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:06.171284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.171244    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:06.171284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.171274    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:06.171284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.171285    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvrnc\" (UniqueName: \"kubernetes.io/projected/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4-kube-api-access-vvrnc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:06.405714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.405680    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"
Nov 02 15:46:06.406181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.405724    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" event={"ID":"03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4","Type":"ContainerDied","Data":"734c026fd64498e476f31df357c14cc3734b6b1169a84fe53358f50523224953"}
Nov 02 15:46:06.406181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.405792    7019 scope.go:122] "RemoveContainer" containerID="047da94fe54fd066472ac19cecfb794614adb064c24c6326e6ed7a9906422db9"
Nov 02 15:46:06.486027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.485993    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"]
Nov 02 15:46:06.489719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.489689    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676"]
Nov 02 15:46:06.600243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.600205    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:06.769507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.769428    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:06.895416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:06.895360    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-fda646/sub-test-model-fda646-kserve-78d5bf6579-vs676" podUID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.215:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:46:07.104209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:07.104114    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:07.367722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:07.348619    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:07.525628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:07.524947    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:07.936459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:07.936370    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" path="/var/lib/kubelet/pods/03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4/volumes"
Nov 02 15:46:08.567329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.567290    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:08.717124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.717087    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:08.774224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.774186    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"]
Nov 02 15:46:08.775311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.775284    7019 cpu_manager.go:414] "containerMap: removing container" podUID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" containerName="main"
Nov 02 15:46:08.775311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.775310    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" containerName="main"
Nov 02 15:46:08.775538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.775327    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" containerName="revoke-keys"
Nov 02 15:46:08.775538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.775335    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" containerName="revoke-keys"
Nov 02 15:46:08.775703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.775543    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fd0c4c-f425-4245-a6a2-3b9af4cd3fb4" containerName="main"
Nov 02 15:46:08.775703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.775562    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" containerName="revoke-keys"
Nov 02 15:46:08.781174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.781152    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.802632 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.802600    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"]
Nov 02 15:46:08.846771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.843610    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 15:46:08.846771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.843914    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 15:46:08.846771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.843953    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 15:46:08.846771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.844174    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rcl8t\""
Nov 02 15:46:08.891064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.891029    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.891244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.891150    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhgt\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-kube-api-access-7zhgt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.891314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.891243    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.891419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.891372    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.891491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.891477    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.891579 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.891564    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.891690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.891668    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c510c41-3a9e-441b-8114-83fc2bf2651c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.993742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.993701    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhgt\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-kube-api-access-7zhgt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.993955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.993874    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.993955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.993939    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.994106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.993989    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.994106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.994058    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.994215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.994149    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c510c41-3a9e-441b-8114-83fc2bf2651c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.994274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.994245    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.995431 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.995404    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.997699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.997663    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.997860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.997790    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:08.999927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:08.999900    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c510c41-3a9e-441b-8114-83fc2bf2651c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:09.000206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.000183    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:09.021186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.021157    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhgt\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-kube-api-access-7zhgt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:09.021457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.021433    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-j7snh\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:09.101723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.096860    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:09.513841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.513794    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:09.513952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.513884    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:46:09.515030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.515007    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"]
Nov 02 15:46:09.613870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.612663    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh" event={"ID":"5c510c41-3a9e-441b-8114-83fc2bf2651c","Type":"ContainerStarted","Data":"492c327bbbb7f6514570ed1707e59fb1e93f3143d926a98e3f66ad042f4ba14f"}
Nov 02 15:46:09.613870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.613731    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:09.668038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:09.667974    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh" podStartSLOduration=1.6679540670000002 podStartE2EDuration="1.667954067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:09.663022395 +0000 UTC m=+2963.916976018" watchObservedRunningTime="2026-11-02 15:46:09.667954067 +0000 UTC m=+2963.921907692"
Nov 02 15:46:10.222342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:10.214945    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:10.626809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:10.626153    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh" event={"ID":"5c510c41-3a9e-441b-8114-83fc2bf2651c","Type":"ContainerStarted","Data":"36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09"}
Nov 02 15:46:10.635670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:10.629790    7019 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-j7snh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:46:10.635670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:10.629849    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh" podUID="5c510c41-3a9e-441b-8114-83fc2bf2651c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:46:11.429593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:11.429555    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:11.571583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:11.571535    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:11.644523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:11.644493    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:46:12.063656 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.063619    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:12.211021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.210987    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:12.345144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.345059    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"]
Nov 02 15:46:12.345367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.345339    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy" containerID="cri-o://48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943" gracePeriod=30
Nov 02 15:46:12.379482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.362728    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"]
Nov 02 15:46:12.379482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.362980    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy" containerID="cri-o://448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4" gracePeriod=30
Nov 02 15:46:12.392066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.392033    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"]
Nov 02 15:46:12.392275 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.392252    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" containerID="cri-o://25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384" gracePeriod=30
Nov 02 15:46:12.408033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.408002    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:12.825592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.825555    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:46:12.825996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.825607    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:46:12.929045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.929002    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:46:12.929281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:12.929086    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:46:13.314041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.314003    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"]
Nov 02 15:46:13.316298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.316270    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:13.319211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.319193    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.338732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.338703    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:46:13.338877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.338819    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:46:13.339246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.339226    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"]
Nov 02 15:46:13.392802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.392769    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.393034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.393018    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.393086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.393049    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxr4\" (UniqueName: \"kubernetes.io/projected/4a936cfe-2f80-4801-ba4c-4b3a9808236e-kube-api-access-pfxr4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.481838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.481801    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:13.495768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.495731    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.495962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.495787    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxr4\" (UniqueName: \"kubernetes.io/projected/4a936cfe-2f80-4801-ba4c-4b3a9808236e-kube-api-access-pfxr4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.495962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.495940    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.497239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.497213    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.503053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.503026    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.538807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.538771    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxr4\" (UniqueName: \"kubernetes.io/projected/4a936cfe-2f80-4801-ba4c-4b3a9808236e-kube-api-access-pfxr4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.714671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.705223    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:13.852025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:13.850195    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:14.153356 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:46:14.153324    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a936cfe_2f80_4801_ba4c_4b3a9808236e.slice/crio-9709416efa0d90bfa9ea173f81e0beec33c5883b5891231bfcfe9b966b206020 WatchSource:0}: Error finding container 9709416efa0d90bfa9ea173f81e0beec33c5883b5891231bfcfe9b966b206020: Status 404 returned error can't find the container with id 9709416efa0d90bfa9ea173f81e0beec33c5883b5891231bfcfe9b966b206020
Nov 02 15:46:14.155547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:14.155518    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"]
Nov 02 15:46:14.546533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:14.546502    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:14.867182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:14.867142    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb" event={"ID":"4a936cfe-2f80-4801-ba4c-4b3a9808236e","Type":"ContainerStarted","Data":"f3660d9e5009f185cafc4acd850d995d21e40d9df13b058f82ddf33661913959"}
Nov 02 15:46:14.867780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:14.867207    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb" event={"ID":"4a936cfe-2f80-4801-ba4c-4b3a9808236e","Type":"ContainerStarted","Data":"9709416efa0d90bfa9ea173f81e0beec33c5883b5891231bfcfe9b966b206020"}
Nov 02 15:46:14.914989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:14.914940    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb" podStartSLOduration=1.914924648 podStartE2EDuration="1.914924648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:14.914919144 +0000 UTC m=+2969.168872773" watchObservedRunningTime="2026-11-02 15:46:14.914924648 +0000 UTC m=+2969.168878268"
Nov 02 15:46:15.879702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:15.879662    7019 generic.go:356] "Generic (PLEG): container finished" podID="4a936cfe-2f80-4801-ba4c-4b3a9808236e" containerID="f3660d9e5009f185cafc4acd850d995d21e40d9df13b058f82ddf33661913959" exitCode=0
Nov 02 15:46:15.880100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:15.879754    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb" event={"ID":"4a936cfe-2f80-4801-ba4c-4b3a9808236e","Type":"ContainerDied","Data":"f3660d9e5009f185cafc4acd850d995d21e40d9df13b058f82ddf33661913959"}
Nov 02 15:46:16.176185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:16.176099    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:16.314002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:16.313962    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"]
Nov 02 15:46:16.375365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:16.375323    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:17.074415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.074379    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:17.240010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.239933    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-openshift-ca\") pod \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") "
Nov 02 15:46:17.240010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.239994    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-xks-ca\") pod \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") "
Nov 02 15:46:17.240210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.240026    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a936cfe-2f80-4801-ba4c-4b3a9808236e-kube-api-access-pfxr4\" (UniqueName: \"kubernetes.io/projected/4a936cfe-2f80-4801-ba4c-4b3a9808236e-kube-api-access-pfxr4\") pod \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\" (UID: \"4a936cfe-2f80-4801-ba4c-4b3a9808236e\") "
Nov 02 15:46:17.240354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.240325    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-openshift-ca" pod "4a936cfe-2f80-4801-ba4c-4b3a9808236e" (UID: "4a936cfe-2f80-4801-ba4c-4b3a9808236e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.240903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.240879    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.242176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.242149    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-xks-ca" pod "4a936cfe-2f80-4801-ba4c-4b3a9808236e" (UID: "4a936cfe-2f80-4801-ba4c-4b3a9808236e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:17.242298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.242285    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a936cfe-2f80-4801-ba4c-4b3a9808236e-kube-api-access-pfxr4" pod "4a936cfe-2f80-4801-ba4c-4b3a9808236e" (UID: "4a936cfe-2f80-4801-ba4c-4b3a9808236e"). InnerVolumeSpecName "kube-api-access-pfxr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.342496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.342462    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a936cfe-2f80-4801-ba4c-4b3a9808236e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.342496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.342495    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxr4\" (UniqueName: \"kubernetes.io/projected/4a936cfe-2f80-4801-ba4c-4b3a9808236e-kube-api-access-pfxr4\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.725445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.725116    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752622    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-data\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752722    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-envoy\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752804    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-credential-socket\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752846    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-kube-api-access-t5rqr\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-kube-api-access-t5rqr\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752878    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-certs\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752906    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-token\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-token\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752931    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-socket\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.753841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.752964    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-podinfo\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.756178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.756147    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-data" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.756369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.756339    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-certs" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.756455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.756436    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-credential-socket" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.756917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.756715    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-socket" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.784909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.775989    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-podinfo" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.788770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.788726    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-envoy" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.821534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.821316    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-kube-api-access-t5rqr" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "kube-api-access-t5rqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.823679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.823636    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-token" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.845589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.840753    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:46:17.854252 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854195    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-credential-socket\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.854516 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854490    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-credential-socket" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.854587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854498    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-envoy\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.854587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854564    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-socket\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.854753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854695    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24bff311-0d0f-423b-afd4-a80671e90fde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bff311-0d0f-423b-afd4-a80671e90fde-istio-podinfo\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.854753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854736    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-istio-token\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-istio-token\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.854869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854774    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-kube-api-access-5cxlz\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-kube-api-access-5cxlz\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.854928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854904    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-data\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.855003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.854986    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24bff311-0d0f-423b-afd4-a80671e90fde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bff311-0d0f-423b-afd4-a80671e90fde-istiod-ca-cert\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.855060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.855025    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-certs\") pod \"24bff311-0d0f-423b-afd4-a80671e90fde\" (UID: \"24bff311-0d0f-423b-afd4-a80671e90fde\") "
Nov 02 15:46:17.855060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.855052    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc7ae13-d1f3-416b-92bd-4bec39365f43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc7ae13-d1f3-416b-92bd-4bec39365f43-istiod-ca-cert\") pod \"acc7ae13-d1f3-416b-92bd-4bec39365f43\" (UID: \"acc7ae13-d1f3-416b-92bd-4bec39365f43\") "
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856218    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856244    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5rqr\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-kube-api-access-t5rqr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856258    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856271    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856283    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856296    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856309    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856322    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.856672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.856334    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc7ae13-d1f3-416b-92bd-4bec39365f43-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.857419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.857239    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-data" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.859597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.857692    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-socket" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.861115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.861079    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-certs" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.862838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.862812    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-envoy" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.864186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.864091    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc7ae13-d1f3-416b-92bd-4bec39365f43-istiod-ca-cert" pod "acc7ae13-d1f3-416b-92bd-4bec39365f43" (UID: "acc7ae13-d1f3-416b-92bd-4bec39365f43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.864600 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.864557    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bff311-0d0f-423b-afd4-a80671e90fde-istiod-ca-cert" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.865201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.865174    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-istio-token" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.865294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.865216    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-kube-api-access-5cxlz" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "kube-api-access-5cxlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.869934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.869905    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24bff311-0d0f-423b-afd4-a80671e90fde-istio-podinfo" pod "24bff311-0d0f-423b-afd4-a80671e90fde" (UID: "24bff311-0d0f-423b-afd4-a80671e90fde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.918821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.918784    7019 generic.go:356] "Generic (PLEG): container finished" podID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerID="448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4" exitCode=0
Nov 02 15:46:17.918956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.918874    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"
Nov 02 15:46:17.919125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.919097    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" event={"ID":"acc7ae13-d1f3-416b-92bd-4bec39365f43","Type":"ContainerDied","Data":"448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4"}
Nov 02 15:46:17.919222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.919159    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929" event={"ID":"acc7ae13-d1f3-416b-92bd-4bec39365f43","Type":"ContainerDied","Data":"5ea3b95570c6debbec3e0177282f7a90296beb3559b19bb85b7721d131e90ecd"}
Nov 02 15:46:17.919222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.919193    7019 scope.go:122] "RemoveContainer" containerID="448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4"
Nov 02 15:46:17.921477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.921453    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:46:17.922194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.922061    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb" event={"ID":"4a936cfe-2f80-4801-ba4c-4b3a9808236e","Type":"ContainerDied","Data":"9709416efa0d90bfa9ea173f81e0beec33c5883b5891231bfcfe9b966b206020"}
Nov 02 15:46:17.922301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.922229    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9709416efa0d90bfa9ea173f81e0beec33c5883b5891231bfcfe9b966b206020"
Nov 02 15:46:17.922366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.922318    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"
Nov 02 15:46:17.923788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.923767    7019 generic.go:356] "Generic (PLEG): container finished" podID="24bff311-0d0f-423b-afd4-a80671e90fde" containerID="25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384" exitCode=0
Nov 02 15:46:17.923893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.923831    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" event={"ID":"24bff311-0d0f-423b-afd4-a80671e90fde","Type":"ContainerDied","Data":"25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384"}
Nov 02 15:46:17.923893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.923858    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn" event={"ID":"24bff311-0d0f-423b-afd4-a80671e90fde","Type":"ContainerDied","Data":"7639d4a2897c8fac8402c18a288ca909d108ec07ed2a64ec16b29d01836b1924"}
Nov 02 15:46:17.924014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.923908    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"
Nov 02 15:46:17.926768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.926748    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"
Nov 02 15:46:17.926905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.926640    7019 generic.go:356] "Generic (PLEG): container finished" podID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerID="48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943" exitCode=0
Nov 02 15:46:17.926976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.926942    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" event={"ID":"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8","Type":"ContainerDied","Data":"48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943"}
Nov 02 15:46:17.927028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.926978    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj" event={"ID":"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8","Type":"ContainerDied","Data":"5807f6460f53bbac5d2956fca93f40203116bd2532d193b8d3d0fac68f48a5dd"}
Nov 02 15:46:17.940891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.936006    7019 scope.go:122] "RemoveContainer" containerID="448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4"
Nov 02 15:46:17.940891 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:46:17.936303    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4\": container with ID starting with 448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4 not found: ID does not exist" containerID="448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4"
Nov 02 15:46:17.940891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.936335    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4"} err="failed to get container status \"448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4\": rpc error: code = NotFound desc = could not find container \"448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4\": container with ID starting with 448552bb41c0c92569951e46dff032ae10064e8211a60397108e1e61738c7cf4 not found: ID does not exist"
Nov 02 15:46:17.940891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.936358    7019 scope.go:122] "RemoveContainer" containerID="25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384"
Nov 02 15:46:17.953836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.953761    7019 scope.go:122] "RemoveContainer" containerID="25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384"
Nov 02 15:46:17.954379 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:46:17.954346    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384\": container with ID starting with 25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384 not found: ID does not exist" containerID="25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384"
Nov 02 15:46:17.954483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.954417    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384"} err="failed to get container status \"25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384\": rpc error: code = NotFound desc = could not find container \"25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384\": container with ID starting with 25d5deac41509638460c4a119d3ba5d68abb34fdb99cded2890f8ced2e3ed384 not found: ID does not exist"
Nov 02 15:46:17.954483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.954442    7019 scope.go:122] "RemoveContainer" containerID="48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943"
Nov 02 15:46:17.958141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.957768    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-podinfo\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.958391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.958374    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-kube-api-access-5k42q\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-kube-api-access-5k42q\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.959336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.959317    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-socket\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.960341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.959734    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-socket" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.960341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.959854    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-podinfo" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.961748    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-token\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-token\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.961849    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istiod-ca-cert\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.961911    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-certs\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.961955    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-data\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962039    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-envoy\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962134    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-credential-socket\") pod \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\" (UID: \"1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8\") "
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962191    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istiod-ca-cert" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962365    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-data" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.962681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962502    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-certs" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962870    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cxlz\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-kube-api-access-5cxlz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962885    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962894    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962903    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bff311-0d0f-423b-afd4-a80671e90fde-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962911    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962920    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc7ae13-d1f3-416b-92bd-4bec39365f43-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962928    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962935    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962944    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962959    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962978    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bff311-0d0f-423b-afd4-a80671e90fde-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962979    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-credential-socket" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.962992    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bff311-0d0f-423b-afd4-a80671e90fde-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.963006    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bff311-0d0f-423b-afd4-a80671e90fde-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.963219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.963018    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:17.967925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.964414    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-envoy" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:17.967925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.967850    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-token" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.969349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.969324    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-kube-api-access-5k42q" pod "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" (UID: "1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8"). InnerVolumeSpecName "kube-api-access-5k42q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:17.977337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.977319    7019 scope.go:122] "RemoveContainer" containerID="48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943"
Nov 02 15:46:17.977723 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:46:17.977694    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943\": container with ID starting with 48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943 not found: ID does not exist" containerID="48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943"
Nov 02 15:46:17.977828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.977734    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943"} err="failed to get container status \"48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943\": rpc error: code = NotFound desc = could not find container \"48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943\": container with ID starting with 48b8554d9c136ba37f688386dd57d2390290f901300d4d7f230b39ef18e79943 not found: ID does not exist"
Nov 02 15:46:17.989977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.989943    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"]
Nov 02 15:46:17.995413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:17.995392    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-8w929"]
Nov 02 15:46:18.032304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.032231    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"]
Nov 02 15:46:18.048032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.036034    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:18.048032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.041020    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-sdssn"]
Nov 02 15:46:18.068168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.068131    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.068168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.068172    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.068368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.068187    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.068368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.068202    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k42q\" (UniqueName: \"kubernetes.io/projected/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8-kube-api-access-5k42q\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:18.308334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.308209    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"]
Nov 02 15:46:18.333935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:18.323138    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3805b8-openshift-default-6c9f89869-d6jbj"]
Nov 02 15:46:19.864060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:19.864019    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" path="/var/lib/kubelet/pods/1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8/volumes"
Nov 02 15:46:19.864797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:19.864776    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" path="/var/lib/kubelet/pods/24bff311-0d0f-423b-afd4-a80671e90fde/volumes"
Nov 02 15:46:19.865334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:19.865313    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" path="/var/lib/kubelet/pods/acc7ae13-d1f3-416b-92bd-4bec39365f43/volumes"
Nov 02 15:46:20.086350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:20.086319    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:21.686508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:21.686244    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:22.710539 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:46:22.710498    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:46:23.238452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:23.238411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:23.906704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:23.906194    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:24.406906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:24.406866    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:24.952514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:24.952471    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:25.134671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:25.132353    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:26.836960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:26.836931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:28.550825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:28.550779    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:28.697295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:28.697246    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:30.351615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:30.351586    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:30.499529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:30.499475    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:30.843794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:30.843242    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:31.349165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.349124    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm" podUID="32f7da44-9c7b-42fc-8b24-89f61b2df9e7" containerName="main" containerID="cri-o://bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789" gracePeriod=45
Nov 02 15:46:31.647080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.647052    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:46:31.729470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.729430    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tls-certs\") pod \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") "
Nov 02 15:46:31.729690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.729508    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-model-cache\") pod \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") "
Nov 02 15:46:31.729835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.729813    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-home\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-home\") pod \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") "
Nov 02 15:46:31.730170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.730118    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-home" pod "32f7da44-9c7b-42fc-8b24-89f61b2df9e7" (UID: "32f7da44-9c7b-42fc-8b24-89f61b2df9e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:31.730170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.730137    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-model-cache" pod "32f7da44-9c7b-42fc-8b24-89f61b2df9e7" (UID: "32f7da44-9c7b-42fc-8b24-89f61b2df9e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:31.730505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.730489    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:31.730586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.730511    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:31.738065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.738034    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tls-certs" pod "32f7da44-9c7b-42fc-8b24-89f61b2df9e7" (UID: "32f7da44-9c7b-42fc-8b24-89f61b2df9e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:46:31.832012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.831978    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-kube-api-access-4j9cl\" (UniqueName: \"kubernetes.io/projected/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-kube-api-access-4j9cl\") pod \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") "
Nov 02 15:46:31.832211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.832078    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tmp-dir\") pod \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") "
Nov 02 15:46:31.832365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.832345    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-dshm\") pod \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\" (UID: \"32f7da44-9c7b-42fc-8b24-89f61b2df9e7\") "
Nov 02 15:46:31.832484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.832345    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tmp-dir" pod "32f7da44-9c7b-42fc-8b24-89f61b2df9e7" (UID: "32f7da44-9c7b-42fc-8b24-89f61b2df9e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:31.832947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.832925    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:31.832947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.832945    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:31.834289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.834261    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-dshm" pod "32f7da44-9c7b-42fc-8b24-89f61b2df9e7" (UID: "32f7da44-9c7b-42fc-8b24-89f61b2df9e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:46:31.834536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.834508    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-kube-api-access-4j9cl" pod "32f7da44-9c7b-42fc-8b24-89f61b2df9e7" (UID: "32f7da44-9c7b-42fc-8b24-89f61b2df9e7"). InnerVolumeSpecName "kube-api-access-4j9cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:46:31.934903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.934866    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:31.934903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:31.934898    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j9cl\" (UniqueName: \"kubernetes.io/projected/32f7da44-9c7b-42fc-8b24-89f61b2df9e7-kube-api-access-4j9cl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:46:32.070542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.070507    7019 generic.go:356] "Generic (PLEG): container finished" podID="32f7da44-9c7b-42fc-8b24-89f61b2df9e7" containerID="bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789" exitCode=0
Nov 02 15:46:32.070752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.070550    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"
Nov 02 15:46:32.070752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.070591    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm" event={"ID":"32f7da44-9c7b-42fc-8b24-89f61b2df9e7","Type":"ContainerDied","Data":"bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789"}
Nov 02 15:46:32.070752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.070643    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm" event={"ID":"32f7da44-9c7b-42fc-8b24-89f61b2df9e7","Type":"ContainerDied","Data":"29e9c027efd13d2212c5c82afded158ddc6a60f6424d5094a393c96082c9d5c4"}
Nov 02 15:46:32.070752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.070703    7019 scope.go:122] "RemoveContainer" containerID="bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789"
Nov 02 15:46:32.085890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.083280    7019 scope.go:122] "RemoveContainer" containerID="bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789"
Nov 02 15:46:32.086333 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:46:32.086310    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789\": container with ID starting with bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789 not found: ID does not exist" containerID="bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789"
Nov 02 15:46:32.086417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.086343    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789"} err="failed to get container status \"bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789\": rpc error: code = NotFound desc = could not find container \"bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789\": container with ID starting with bd54b46c023eab376661329089fd89604c8dd831d6604c397da4f758fdb9b789 not found: ID does not exist"
Nov 02 15:46:32.134174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.134110    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"]
Nov 02 15:46:32.141018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.140992    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6f2e0c/sub-test-model-6f2e0c-kserve-75d6b86489-92qkm"]
Nov 02 15:46:32.154381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:32.154358    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:33.785394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:33.785359    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:33.864949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:33.864911    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f7da44-9c7b-42fc-8b24-89f61b2df9e7" path="/var/lib/kubelet/pods/32f7da44-9c7b-42fc-8b24-89f61b2df9e7/volumes"
Nov 02 15:46:35.349336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:35.349303    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:36.933659 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:36.933621    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:38.685625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:38.685583    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:38.840664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:38.840590    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:39.004941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:39.004865    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:40.575210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:40.575170    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:42.278667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:42.278487    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:43.905900 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:43.905863    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:45.594935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:45.594897    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:46.976798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:46.976766    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:47.525670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:47.524889    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:47.593344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:47.593301    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"]
Nov 02 15:46:47.607838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:47.607802    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-501738-bzzdb"]
Nov 02 15:46:47.676743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:47.676708    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:47.864025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:47.863991    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a936cfe-2f80-4801-ba4c-4b3a9808236e" path="/var/lib/kubelet/pods/4a936cfe-2f80-4801-ba4c-4b3a9808236e/volumes"
Nov 02 15:46:48.093039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:48.093005    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:49.508432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:49.508396    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:49.561858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:49.561830    7019 kubelet.go:1689] "Image garbage collection succeeded"
Nov 02 15:46:50.184999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:50.184968    7019 scope.go:122] "RemoveContainer" containerID="48ced2708b8d49064c9503a6dd99ca723a0042cdcc9450dd5abab2349256ae5e"
Nov 02 15:46:50.199040 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:50.199005    7019 scope.go:122] "RemoveContainer" containerID="664b1c66b3b3c3bd6e10f107ef280ead978f93b35c8e904e7229842b0fa59570"
Nov 02 15:46:50.211947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:50.211919    7019 scope.go:122] "RemoveContainer" containerID="5d36fd6cfe0f0bc929592e3b3883412ec5bf74599bec25731e34ce3c5f15798e"
Nov 02 15:46:50.223986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:50.223963    7019 scope.go:122] "RemoveContainer" containerID="95905fc27d55526d0688d8a5aeee9772e96e8eba6d878731b5fc3e5ee2b6a889"
Nov 02 15:46:50.311880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:50.311848    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:51.087255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:51.087216    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:52.613718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:52.613680    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:52.711794 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:46:52.711752    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:46:53.016508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:53.016420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:53.171032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:53.171003    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:54.340719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:54.340683    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:54.499320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:54.499281    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:54.747062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:54.747023    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:46:55.316133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:55.315998    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:55.541818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:55.541770    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:55.765082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:55.765049    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:56.366485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.366447    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"]
Nov 02 15:46:56.367179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367163    7019 cpu_manager.go:414] "containerMap: removing container" podUID="32f7da44-9c7b-42fc-8b24-89f61b2df9e7" containerName="main"
Nov 02 15:46:56.367234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367181    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f7da44-9c7b-42fc-8b24-89f61b2df9e7" containerName="main"
Nov 02 15:46:56.367294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367246    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4a936cfe-2f80-4801-ba4c-4b3a9808236e" containerName="revoke-keys"
Nov 02 15:46:56.367294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367252    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a936cfe-2f80-4801-ba4c-4b3a9808236e" containerName="revoke-keys"
Nov 02 15:46:56.367294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367267    7019 cpu_manager.go:414] "containerMap: removing container" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy"
Nov 02 15:46:56.367294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367273    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367296    7019 cpu_manager.go:414] "containerMap: removing container" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367303    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367315    7019 cpu_manager.go:414] "containerMap: removing container" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367320    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367441    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc7ae13-d1f3-416b-92bd-4bec39365f43" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367458    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="24bff311-0d0f-423b-afd4-a80671e90fde" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367467    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc5a31d-14c9-4d57-acd3-e29a49a7f4c8" containerName="istio-proxy"
Nov 02 15:46:56.367485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367474    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f7da44-9c7b-42fc-8b24-89f61b2df9e7" containerName="main"
Nov 02 15:46:56.367845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.367494    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a936cfe-2f80-4801-ba4c-4b3a9808236e" containerName="revoke-keys"
Nov 02 15:46:56.373739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.373720    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.389898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.389865    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"]
Nov 02 15:46:56.422735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.422699    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:56.454641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.454612    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:46:56.454938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.454920    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:46:56.525400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.525368    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.525581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.525514    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.525581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.525554    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjllv\" (UniqueName: \"kubernetes.io/projected/e015b9cc-c665-4488-a2bd-6c576c341859-kube-api-access-gjllv\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.626952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.626868    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.627413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.626998    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.627413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.627047    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjllv\" (UniqueName: \"kubernetes.io/projected/e015b9cc-c665-4488-a2bd-6c576c341859-kube-api-access-gjllv\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.627806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.627780    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.629416 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.629396    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.653382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.653356    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjllv\" (UniqueName: \"kubernetes.io/projected/e015b9cc-c665-4488-a2bd-6c576c341859-kube-api-access-gjllv\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:56.689961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:56.689932    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:46:57.206581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:57.206539    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"]
Nov 02 15:46:57.216348 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:46:57.211507    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode015b9cc_c665_4488_a2bd_6c576c341859.slice/crio-b838d479351e9e257cc4c9f0aa7bd7b3f22786906cb1de160bb8a96be697b306 WatchSource:0}: Error finding container b838d479351e9e257cc4c9f0aa7bd7b3f22786906cb1de160bb8a96be697b306: Status 404 returned error can't find the container with id b838d479351e9e257cc4c9f0aa7bd7b3f22786906cb1de160bb8a96be697b306
Nov 02 15:46:57.360545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:57.360509    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2" event={"ID":"e015b9cc-c665-4488-a2bd-6c576c341859","Type":"ContainerStarted","Data":"b838d479351e9e257cc4c9f0aa7bd7b3f22786906cb1de160bb8a96be697b306"}
Nov 02 15:46:57.955494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:57.955452    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:58.137603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:58.128283    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:58.291588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:58.291461    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:58.606297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:58.604502    7019 generic.go:356] "Generic (PLEG): container finished" podID="e015b9cc-c665-4488-a2bd-6c576c341859" containerID="cefcdfee4b43a6fc333f570ae704fca0f9ffc2afb4394c2c5c20162e8d39228c" exitCode=0
Nov 02 15:46:58.606297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:58.604549    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2" event={"ID":"e015b9cc-c665-4488-a2bd-6c576c341859","Type":"ContainerDied","Data":"cefcdfee4b43a6fc333f570ae704fca0f9ffc2afb4394c2c5c20162e8d39228c"}
Nov 02 15:46:58.642875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:58.642762    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:58.798849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:58.798818    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:59.071355 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:59.071313    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:59.279842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:59.279806    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:46:59.915413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:46:59.915333    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:47:00.008336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.008289    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e015b9cc-c665-4488-a2bd-6c576c341859-kube-api-access-gjllv\" (UniqueName: \"kubernetes.io/projected/e015b9cc-c665-4488-a2bd-6c576c341859-kube-api-access-gjllv\") pod \"e015b9cc-c665-4488-a2bd-6c576c341859\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") "
Nov 02 15:47:00.008548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.008466    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-xks-ca\") pod \"e015b9cc-c665-4488-a2bd-6c576c341859\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") "
Nov 02 15:47:00.008702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.008683    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-openshift-ca\") pod \"e015b9cc-c665-4488-a2bd-6c576c341859\" (UID: \"e015b9cc-c665-4488-a2bd-6c576c341859\") "
Nov 02 15:47:00.016300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.016269    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-openshift-ca" pod "e015b9cc-c665-4488-a2bd-6c576c341859" (UID: "e015b9cc-c665-4488-a2bd-6c576c341859"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:00.021547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.019868    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e015b9cc-c665-4488-a2bd-6c576c341859-kube-api-access-gjllv" pod "e015b9cc-c665-4488-a2bd-6c576c341859" (UID: "e015b9cc-c665-4488-a2bd-6c576c341859"). InnerVolumeSpecName "kube-api-access-gjllv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:00.023296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.023262    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-xks-ca" pod "e015b9cc-c665-4488-a2bd-6c576c341859" (UID: "e015b9cc-c665-4488-a2bd-6c576c341859"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:00.111371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.111332    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.111371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.111371    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjllv\" (UniqueName: \"kubernetes.io/projected/e015b9cc-c665-4488-a2bd-6c576c341859-kube-api-access-gjllv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.111800 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.111385    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e015b9cc-c665-4488-a2bd-6c576c341859-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:00.511813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.511773    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:00.627716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.627463    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2" event={"ID":"e015b9cc-c665-4488-a2bd-6c576c341859","Type":"ContainerDied","Data":"b838d479351e9e257cc4c9f0aa7bd7b3f22786906cb1de160bb8a96be697b306"}
Nov 02 15:47:00.627716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.627493    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"
Nov 02 15:47:00.627716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:00.627571    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b838d479351e9e257cc4c9f0aa7bd7b3f22786906cb1de160bb8a96be697b306"
Nov 02 15:47:02.126839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.126802    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:02.895582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.895540    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"]
Nov 02 15:47:02.896883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.896851    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e015b9cc-c665-4488-a2bd-6c576c341859" containerName="revoke-keys"
Nov 02 15:47:02.896883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.896879    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e015b9cc-c665-4488-a2bd-6c576c341859" containerName="revoke-keys"
Nov 02 15:47:02.897115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.897090    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e015b9cc-c665-4488-a2bd-6c576c341859" containerName="revoke-keys"
Nov 02 15:47:02.901070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.901049    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:02.909931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.909903    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"]
Nov 02 15:47:02.924979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.924955    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:47:02.937593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.937570    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:47:02.960134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.960103    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:02.960311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.960236    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:02.960802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:02.960537    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4pns\" (UniqueName: \"kubernetes.io/projected/55c75944-01cb-423c-bfae-0046bee77d68-kube-api-access-r4pns\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.062276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.062238    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.062456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.062303    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.062456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.062430    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4pns\" (UniqueName: \"kubernetes.io/projected/55c75944-01cb-423c-bfae-0046bee77d68-kube-api-access-r4pns\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.063071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.063046    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.064871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.064845    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.096420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.096378    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4pns\" (UniqueName: \"kubernetes.io/projected/55c75944-01cb-423c-bfae-0046bee77d68-kube-api-access-r4pns\") pod \"maas-api-revoke-sub-46794ced-wdvtd\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.221956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.221875    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:03.561499 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:47:03.561470    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c75944_01cb_423c_bfae_0046bee77d68.slice/crio-dfa14f23c546b8af22a0a44332ced6fcc798706470fac95a2e587b47a6eb75c5 WatchSource:0}: Error finding container dfa14f23c546b8af22a0a44332ced6fcc798706470fac95a2e587b47a6eb75c5: Status 404 returned error can't find the container with id dfa14f23c546b8af22a0a44332ced6fcc798706470fac95a2e587b47a6eb75c5
Nov 02 15:47:03.563173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.563149    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"]
Nov 02 15:47:03.652408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:03.652369    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd" event={"ID":"55c75944-01cb-423c-bfae-0046bee77d68","Type":"ContainerStarted","Data":"dfa14f23c546b8af22a0a44332ced6fcc798706470fac95a2e587b47a6eb75c5"}
Nov 02 15:47:04.436440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:04.433246    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:04.671690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:04.671468    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd" event={"ID":"55c75944-01cb-423c-bfae-0046bee77d68","Type":"ContainerStarted","Data":"8ec257e3628bd0598c10ef934e39245e404a6e1b59d01e5c44acb94f88307302"}
Nov 02 15:47:04.729266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:04.729140    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd" podStartSLOduration=2.7291260250000002 podStartE2EDuration="2.729126025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:04.727241808 +0000 UTC m=+3018.981195431" watchObservedRunningTime="2026-11-02 15:47:04.729126025 +0000 UTC m=+3018.983079649"
Nov 02 15:47:05.683581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:05.683542    7019 generic.go:356] "Generic (PLEG): container finished" podID="55c75944-01cb-423c-bfae-0046bee77d68" containerID="8ec257e3628bd0598c10ef934e39245e404a6e1b59d01e5c44acb94f88307302" exitCode=0
Nov 02 15:47:05.684060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:05.683586    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd" event={"ID":"55c75944-01cb-423c-bfae-0046bee77d68","Type":"ContainerDied","Data":"8ec257e3628bd0598c10ef934e39245e404a6e1b59d01e5c44acb94f88307302"}
Nov 02 15:47:06.076153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.076077    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:06.268919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.268879    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:06.407374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.407279    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"]
Nov 02 15:47:06.408391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.408357    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy" containerID="cri-o://2e149037318bcef30d0bf49a8029303425e881ed8e85e2cf39cae706ea532717" gracePeriod=30
Nov 02 15:47:06.414150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.414122    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"]
Nov 02 15:47:06.414359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.414334    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy" containerID="cri-o://87533de51cef7af0570ab422e5c475b73926d5e705be3c312ecd59c4468a315a" gracePeriod=30
Nov 02 15:47:06.420250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.420224    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"]
Nov 02 15:47:06.420466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.420443    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy" containerID="cri-o://601e39cdea53e8623e160c53c487fa70d91369f2a2f999d10d047b46c089cd52" gracePeriod=30
Nov 02 15:47:06.424836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.424814    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"]
Nov 02 15:47:06.424993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.424976    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy" containerID="cri-o://bf83d073a1bfb2514a2540e152340333e142af39557760a353b2026a20f7f6ed" gracePeriod=30
Nov 02 15:47:06.447860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.446949    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:06.893841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.893798    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:06.990739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.990526    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-openshift-ca\") pod \"55c75944-01cb-423c-bfae-0046bee77d68\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") "
Nov 02 15:47:06.990927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.990787    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-xks-ca\") pod \"55c75944-01cb-423c-bfae-0046bee77d68\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") "
Nov 02 15:47:06.990927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.990910    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c75944-01cb-423c-bfae-0046bee77d68-kube-api-access-r4pns\" (UniqueName: \"kubernetes.io/projected/55c75944-01cb-423c-bfae-0046bee77d68-kube-api-access-r4pns\") pod \"55c75944-01cb-423c-bfae-0046bee77d68\" (UID: \"55c75944-01cb-423c-bfae-0046bee77d68\") "
Nov 02 15:47:06.992461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.992430    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-openshift-ca" pod "55c75944-01cb-423c-bfae-0046bee77d68" (UID: "55c75944-01cb-423c-bfae-0046bee77d68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:06.995780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:06.995750    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-xks-ca" pod "55c75944-01cb-423c-bfae-0046bee77d68" (UID: "55c75944-01cb-423c-bfae-0046bee77d68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:07.000079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.000050    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c75944-01cb-423c-bfae-0046bee77d68-kube-api-access-r4pns" pod "55c75944-01cb-423c-bfae-0046bee77d68" (UID: "55c75944-01cb-423c-bfae-0046bee77d68"). InnerVolumeSpecName "kube-api-access-r4pns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:07.093119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.093083    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:07.093119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.093114    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55c75944-01cb-423c-bfae-0046bee77d68-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:07.093119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.093126    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4pns\" (UniqueName: \"kubernetes.io/projected/55c75944-01cb-423c-bfae-0046bee77d68-kube-api-access-r4pns\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:07.464727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.464690    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"]
Nov 02 15:47:07.511898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.511865    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"]
Nov 02 15:47:07.737826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.737702    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd" event={"ID":"55c75944-01cb-423c-bfae-0046bee77d68","Type":"ContainerDied","Data":"dfa14f23c546b8af22a0a44332ced6fcc798706470fac95a2e587b47a6eb75c5"}
Nov 02 15:47:07.738117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.738071    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa14f23c546b8af22a0a44332ced6fcc798706470fac95a2e587b47a6eb75c5"
Nov 02 15:47:07.738383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.738365    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"
Nov 02 15:47:07.778602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.778571    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:07.927479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:07.927438    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:08.259983 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:08.257371    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02302e75_aacc_4653_b806_4f06991d24c3.slice/crio-c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:47:08.438950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.438920    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:08.705466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.705432    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:08.747210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.747177    7019 generic.go:356] "Generic (PLEG): container finished" podID="02302e75-aacc-4653-b806-4f06991d24c3" containerID="c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d" exitCode=0
Nov 02 15:47:08.747435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.747258    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" event={"ID":"02302e75-aacc-4653-b806-4f06991d24c3","Type":"ContainerDied","Data":"c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d"}
Nov 02 15:47:08.747885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.747866    7019 scope.go:122] "RemoveContainer" containerID="c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d"
Nov 02 15:47:08.750780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.750747    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:47:08.750892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.750819    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:47:08.750892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:08.750851    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:47:09.760321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:09.760284    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" event={"ID":"02302e75-aacc-4653-b806-4f06991d24c3","Type":"ContainerStarted","Data":"b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1"}
Nov 02 15:47:09.988677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:09.985988    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:09.998693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:09.997616    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:47:09.998693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:09.998442    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:47:09.998693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:09.998499    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Nov 02 15:47:10.126088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:10.126055    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"]
Nov 02 15:47:10.133515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:10.133478    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-78d075-bphv2"]
Nov 02 15:47:10.148733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:10.148693    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:10.296826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:10.296791    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:10.584726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:10.581226    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:10.996805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:10.996768    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:47:10.997256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:10.996826    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Nov 02 15:47:11.911198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:11.897205    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:11.939767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:11.929665    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e015b9cc-c665-4488-a2bd-6c576c341859" path="/var/lib/kubelet/pods/e015b9cc-c665-4488-a2bd-6c576c341859/volumes"
Nov 02 15:47:12.030994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.030878    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:47:12.042486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.038730    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:47:12.042486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.038834    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Nov 02 15:47:12.042486 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.039555    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:12.154463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.154339    7019 generic.go:356] "Generic (PLEG): container finished" podID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerID="bf83d073a1bfb2514a2540e152340333e142af39557760a353b2026a20f7f6ed" exitCode=0
Nov 02 15:47:12.155322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.155292    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" event={"ID":"dfc1aa65-0e3b-48f6-9dee-756668d230f0","Type":"ContainerDied","Data":"bf83d073a1bfb2514a2540e152340333e142af39557760a353b2026a20f7f6ed"}
Nov 02 15:47:12.159136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.159105    7019 generic.go:356] "Generic (PLEG): container finished" podID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerID="2e149037318bcef30d0bf49a8029303425e881ed8e85e2cf39cae706ea532717" exitCode=0
Nov 02 15:47:12.159359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.159322    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" event={"ID":"68cb0ace-10e5-4702-8d93-029c41e82e1f","Type":"ContainerDied","Data":"2e149037318bcef30d0bf49a8029303425e881ed8e85e2cf39cae706ea532717"}
Nov 02 15:47:12.161262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.161242    7019 generic.go:356] "Generic (PLEG): container finished" podID="a330d41a-4746-4900-af0d-2b9917fb199d" containerID="601e39cdea53e8623e160c53c487fa70d91369f2a2f999d10d047b46c089cd52" exitCode=0
Nov 02 15:47:12.161461 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.161338    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"
Nov 02 15:47:12.161553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.161327    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s" event={"ID":"a330d41a-4746-4900-af0d-2b9917fb199d","Type":"ContainerDied","Data":"601e39cdea53e8623e160c53c487fa70d91369f2a2f999d10d047b46c089cd52"}
Nov 02 15:47:12.161615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.161593    7019 scope.go:122] "RemoveContainer" containerID="601e39cdea53e8623e160c53c487fa70d91369f2a2f999d10d047b46c089cd52"
Nov 02 15:47:12.163008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.162990    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:47:12.165754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.165730    7019 generic.go:356] "Generic (PLEG): container finished" podID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerID="87533de51cef7af0570ab422e5c475b73926d5e705be3c312ecd59c4468a315a" exitCode=0
Nov 02 15:47:12.165925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.165908    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" event={"ID":"4512b064-bd7d-439c-8767-bf7d58c0f2fd","Type":"ContainerDied","Data":"87533de51cef7af0570ab422e5c475b73926d5e705be3c312ecd59c4468a315a"}
Nov 02 15:47:12.180034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.179869    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:47:12.183575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.183536    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:47:12.197867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.197536    7019 scope.go:122] "RemoveContainer" containerID="87533de51cef7af0570ab422e5c475b73926d5e705be3c312ecd59c4468a315a"
Nov 02 15:47:12.199474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.199445    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:12.249811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.249775    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-socket\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.250016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.249830    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-data\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.250016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.249867    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-token\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-token\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.250016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.249911    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-kube-api-access-2cb29\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-kube-api-access-2cb29\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.250016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.249952    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-socket\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.250016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.249994    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-data\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.250305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250076    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-kube-api-access-bpbmx\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-kube-api-access-bpbmx\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.250305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250108    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a330d41a-4746-4900-af0d-2b9917fb199d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a330d41a-4746-4900-af0d-2b9917fb199d-istiod-ca-cert\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.250305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250146    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a330d41a-4746-4900-af0d-2b9917fb199d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a330d41a-4746-4900-af0d-2b9917fb199d-istio-podinfo\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.250305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250182    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-socket\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.250305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250232    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-envoy\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.250305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250295    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-kube-api-access-5mlvf\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-kube-api-access-5mlvf\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250338    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-credential-socket\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250363    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-kube-api-access-7qqqn\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-kube-api-access-7qqqn\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250398    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-credential-socket\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250422    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-credential-socket\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250458    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-podinfo\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250476    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-certs\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250505    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-certs\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250536    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-token\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250575    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-certs\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250592    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-envoy\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250628    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-podinfo\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.250693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250675    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istiod-ca-cert\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250715    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-certs\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250759    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-envoy\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250801    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-podinfo\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250818    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-envoy\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250843    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-socket\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250876    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68cb0ace-10e5-4702-8d93-029c41e82e1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cb0ace-10e5-4702-8d93-029c41e82e1f-istiod-ca-cert\") pod \"68cb0ace-10e5-4702-8d93-029c41e82e1f\" (UID: \"68cb0ace-10e5-4702-8d93-029c41e82e1f\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250912    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-credential-socket\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250933    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-istio-token\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-istio-token\") pod \"a330d41a-4746-4900-af0d-2b9917fb199d\" (UID: \"a330d41a-4746-4900-af0d-2b9917fb199d\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250959    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-token\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-token\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250976    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-data\") pod \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\" (UID: \"4512b064-bd7d-439c-8767-bf7d58c0f2fd\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.251004    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-data\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.251316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.251060    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istiod-ca-cert\") pod \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\" (UID: \"dfc1aa65-0e3b-48f6-9dee-756668d230f0\") "
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250180    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-socket" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250468    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-socket" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.250604    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-data" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.251200    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-certs" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.251378    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-data" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.252443    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-credential-socket" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.252838    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istiod-ca-cert" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.253256    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-credential-socket" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.253343    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-socket" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.253365    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a330d41a-4746-4900-af0d-2b9917fb199d-istiod-ca-cert" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.253389    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-data" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.253599    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-data" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.253749    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68cb0ace-10e5-4702-8d93-029c41e82e1f-istiod-ca-cert" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.255332    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istiod-ca-cert" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.256179    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-certs" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.256769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.256493    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-certs" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.256779    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-certs" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.256855    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-socket" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.256983    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-credential-socket" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.257204    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-credential-socket" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.257810    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-token" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.258271    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-podinfo" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.258332    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-token" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.259065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.258864    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-envoy" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.259620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.259268    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-podinfo" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:12.259620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.259276    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-envoy" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.260702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.260668    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-kube-api-access-2cb29" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "kube-api-access-2cb29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.262611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.262590    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-envoy" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.263961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.263936    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a330d41a-4746-4900-af0d-2b9917fb199d-istio-podinfo" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:12.264061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.263963    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-token" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.264061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.263968    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-kube-api-access-7qqqn" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "kube-api-access-7qqqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.264061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.263990    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-envoy" pod "68cb0ace-10e5-4702-8d93-029c41e82e1f" (UID: "68cb0ace-10e5-4702-8d93-029c41e82e1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:12.264061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.263997    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-kube-api-access-5mlvf" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "kube-api-access-5mlvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.264238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.264116    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-podinfo" pod "dfc1aa65-0e3b-48f6-9dee-756668d230f0" (UID: "dfc1aa65-0e3b-48f6-9dee-756668d230f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:12.264238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.264131    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-kube-api-access-bpbmx" pod "4512b064-bd7d-439c-8767-bf7d58c0f2fd" (UID: "4512b064-bd7d-439c-8767-bf7d58c0f2fd"). InnerVolumeSpecName "kube-api-access-bpbmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.303966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.303860    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-istio-token" pod "a330d41a-4746-4900-af0d-2b9917fb199d" (UID: "a330d41a-4746-4900-af0d-2b9917fb199d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:12.353136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353102    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353134    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353145    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353156    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353168    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353178    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353189    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353200    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cb29\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-kube-api-access-2cb29\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353214    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353225    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353237    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbmx\" (UniqueName: \"kubernetes.io/projected/4512b064-bd7d-439c-8767-bf7d58c0f2fd-kube-api-access-bpbmx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353252    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a330d41a-4746-4900-af0d-2b9917fb199d-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353264    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a330d41a-4746-4900-af0d-2b9917fb199d-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353276    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353287    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353299    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mlvf\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-kube-api-access-5mlvf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353311    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353325    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqqn\" (UniqueName: \"kubernetes.io/projected/68cb0ace-10e5-4702-8d93-029c41e82e1f-kube-api-access-7qqqn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353337    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353353    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353367    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353381    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353393    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353405    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353420    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353432    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353444    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68cb0ace-10e5-4702-8d93-029c41e82e1f-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.353448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353457    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353472    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68cb0ace-10e5-4702-8d93-029c41e82e1f-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353484    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353496    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc1aa65-0e3b-48f6-9dee-756668d230f0-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353508    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a330d41a-4746-4900-af0d-2b9917fb199d-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353522    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc1aa65-0e3b-48f6-9dee-756668d230f0-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353534    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68cb0ace-10e5-4702-8d93-029c41e82e1f-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353548    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4512b064-bd7d-439c-8767-bf7d58c0f2fd-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.354634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.353571    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a330d41a-4746-4900-af0d-2b9917fb199d-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:12.472678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.471741    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" podUID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" containerName="payload-processing" containerID="cri-o://7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d" gracePeriod=25
Nov 02 15:47:12.512806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.512765    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" podUID="d4209db8-105a-451a-9a43-719fe1a2a29b" containerName="payload-pre-processing" containerID="cri-o://17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a" gracePeriod=25
Nov 02 15:47:12.556779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.556674    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"]
Nov 02 15:47:12.563261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.563227    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-bjc5s"]
Nov 02 15:47:12.999754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:12.999714    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:47:13.000057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.000025    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Nov 02 15:47:13.219780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.219748    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"]
Nov 02 15:47:13.220845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.220683    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"
Nov 02 15:47:13.221704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.221671    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy"
Nov 02 15:47:13.221857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.221841    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy"
Nov 02 15:47:13.221995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.221981    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy"
Nov 02 15:47:13.222083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222071    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy"
Nov 02 15:47:13.222235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222204    7019 cpu_manager.go:414] "containerMap: removing container" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy"
Nov 02 15:47:13.222235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222220    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy"
Nov 02 15:47:13.222235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222241    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy"
Nov 02 15:47:13.222495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222249    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy"
Nov 02 15:47:13.222495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222274    7019 cpu_manager.go:414] "containerMap: removing container" podUID="55c75944-01cb-423c-bfae-0046bee77d68" containerName="revoke-keys"
Nov 02 15:47:13.222495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222282    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c75944-01cb-423c-bfae-0046bee77d68" containerName="revoke-keys"
Nov 02 15:47:13.222782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222558    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" containerName="istio-proxy"
Nov 02 15:47:13.222782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222582    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" containerName="istio-proxy"
Nov 02 15:47:13.222782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222600    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" containerName="istio-proxy"
Nov 02 15:47:13.222782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222611    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" containerName="istio-proxy"
Nov 02 15:47:13.222782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.222623    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c75944-01cb-423c-bfae-0046bee77d68" containerName="revoke-keys"
Nov 02 15:47:13.227090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.227064    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"
Nov 02 15:47:13.229265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.229241    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"
Nov 02 15:47:13.229947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.229925    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"]
Nov 02 15:47:13.230058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.229966    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc" event={"ID":"4512b064-bd7d-439c-8767-bf7d58c0f2fd","Type":"ContainerDied","Data":"f473572d219144049708a79af2275850ac95db636fb8d2b4dea37a51668b43e6"}
Nov 02 15:47:13.230058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.230002    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk" event={"ID":"dfc1aa65-0e3b-48f6-9dee-756668d230f0","Type":"ContainerDied","Data":"ab0c9e7c667b54fdf5eaf3d33091247dffb0da8112df88a0b839d80396e07f31"}
Nov 02 15:47:13.230058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.230030    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf" event={"ID":"68cb0ace-10e5-4702-8d93-029c41e82e1f","Type":"ContainerDied","Data":"12db2a994b1527d90e0bc800e6de5bec44bc181334a43960b983425fdf8a2724"}
Nov 02 15:47:13.230258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.230062    7019 scope.go:122] "RemoveContainer" containerID="bf83d073a1bfb2514a2540e152340333e142af39557760a353b2026a20f7f6ed"
Nov 02 15:47:13.230258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.230183    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.248794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.248764    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:47:13.248922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.248839    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:47:13.249591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.249567    7019 scope.go:122] "RemoveContainer" containerID="2e149037318bcef30d0bf49a8029303425e881ed8e85e2cf39cae706ea532717"
Nov 02 15:47:13.305010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.304961    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwq84\" (UniqueName: \"kubernetes.io/projected/501cd36a-77b8-429b-b253-949f798f9a9d-kube-api-access-xwq84\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.305361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.305333    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.305592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.305419    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.335503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.335471    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"]
Nov 02 15:47:13.339226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.339200    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-qwmpc"]
Nov 02 15:47:13.406671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.406622    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwq84\" (UniqueName: \"kubernetes.io/projected/501cd36a-77b8-429b-b253-949f798f9a9d-kube-api-access-xwq84\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.407049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.407021    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.407167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.407090    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.407717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.407692    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.409622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.409596    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.427438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.427401    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"]
Nov 02 15:47:13.431752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.431725    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-66llf"]
Nov 02 15:47:13.433308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.433282    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwq84\" (UniqueName: \"kubernetes.io/projected/501cd36a-77b8-429b-b253-949f798f9a9d-kube-api-access-xwq84\") pod \"maas-api-revoke-sub-7e54315a-4q7cj\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.502925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.502806    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"]
Nov 02 15:47:13.506890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.506837    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e80c77-openshift-default-86b4fdb5b9-8pzdk"]
Nov 02 15:47:13.558065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.558033    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:13.600293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.600256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:13.842052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.841921    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:13.862619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.862585    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4512b064-bd7d-439c-8767-bf7d58c0f2fd" path="/var/lib/kubelet/pods/4512b064-bd7d-439c-8767-bf7d58c0f2fd/volumes"
Nov 02 15:47:13.863180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.863163    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68cb0ace-10e5-4702-8d93-029c41e82e1f" path="/var/lib/kubelet/pods/68cb0ace-10e5-4702-8d93-029c41e82e1f/volumes"
Nov 02 15:47:13.863626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.863612    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a330d41a-4746-4900-af0d-2b9917fb199d" path="/var/lib/kubelet/pods/a330d41a-4746-4900-af0d-2b9917fb199d/volumes"
Nov 02 15:47:13.864409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.864394    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc1aa65-0e3b-48f6-9dee-756668d230f0" path="/var/lib/kubelet/pods/dfc1aa65-0e3b-48f6-9dee-756668d230f0/volumes"
Nov 02 15:47:13.998679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:13.998623    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"]
Nov 02 15:47:14.000875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:14.000051    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body=
Nov 02 15:47:14.000875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:14.000106    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused"
Nov 02 15:47:14.009830 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:47:14.009797    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod501cd36a_77b8_429b_b253_949f798f9a9d.slice/crio-ab92865792d0d73fc866116afcdc195135171eb9ba83169f72edbf36ea4c5920 WatchSource:0}: Error finding container ab92865792d0d73fc866116afcdc195135171eb9ba83169f72edbf36ea4c5920: Status 404 returned error can't find the container with id ab92865792d0d73fc866116afcdc195135171eb9ba83169f72edbf36ea4c5920
Nov 02 15:47:14.247831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:14.247789    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj" event={"ID":"501cd36a-77b8-429b-b253-949f798f9a9d","Type":"ContainerStarted","Data":"ab92865792d0d73fc866116afcdc195135171eb9ba83169f72edbf36ea4c5920"}
Nov 02 15:47:15.076436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.076171    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:47:15.077729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.077195    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:47:15.079613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.079593    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:47:15.192099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.191754    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:15.280310 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.280268    7019 generic.go:356] "Generic (PLEG): container finished" podID="501cd36a-77b8-429b-b253-949f798f9a9d" containerID="9ab755672ea89618021a603763cb9b677bbd2d1b2d5452a37e5ba4447e4b1fea" exitCode=0
Nov 02 15:47:15.282030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.282001    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj" event={"ID":"501cd36a-77b8-429b-b253-949f798f9a9d","Type":"ContainerDied","Data":"9ab755672ea89618021a603763cb9b677bbd2d1b2d5452a37e5ba4447e4b1fea"}
Nov 02 15:47:15.601590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.601539    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" podUID="d4209db8-105a-451a-9a43-719fe1a2a29b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:15.601795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.601630    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" podUID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:15.956503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:15.956461    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:16.103240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.103200    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:16.461935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.461910    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:16.578521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.578425    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-xks-ca\") pod \"501cd36a-77b8-429b-b253-949f798f9a9d\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") "
Nov 02 15:47:16.578706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.578639    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-openshift-ca\") pod \"501cd36a-77b8-429b-b253-949f798f9a9d\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") "
Nov 02 15:47:16.578908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.578892    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501cd36a-77b8-429b-b253-949f798f9a9d-kube-api-access-xwq84\" (UniqueName: \"kubernetes.io/projected/501cd36a-77b8-429b-b253-949f798f9a9d-kube-api-access-xwq84\") pod \"501cd36a-77b8-429b-b253-949f798f9a9d\" (UID: \"501cd36a-77b8-429b-b253-949f798f9a9d\") "
Nov 02 15:47:16.579070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.579043    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-openshift-ca" pod "501cd36a-77b8-429b-b253-949f798f9a9d" (UID: "501cd36a-77b8-429b-b253-949f798f9a9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:16.579588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.579561    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:16.580608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.580580    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-xks-ca" pod "501cd36a-77b8-429b-b253-949f798f9a9d" (UID: "501cd36a-77b8-429b-b253-949f798f9a9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:16.581306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.581259    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501cd36a-77b8-429b-b253-949f798f9a9d-kube-api-access-xwq84" pod "501cd36a-77b8-429b-b253-949f798f9a9d" (UID: "501cd36a-77b8-429b-b253-949f798f9a9d"). InnerVolumeSpecName "kube-api-access-xwq84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:16.681011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.680973    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwq84\" (UniqueName: \"kubernetes.io/projected/501cd36a-77b8-429b-b253-949f798f9a9d-kube-api-access-xwq84\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:16.681011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.681004    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/501cd36a-77b8-429b-b253-949f798f9a9d-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:16.923081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:16.923042    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:17.159670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.159483    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:17.301497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.301405    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"
Nov 02 15:47:17.301497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.301448    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj" event={"ID":"501cd36a-77b8-429b-b253-949f798f9a9d","Type":"ContainerDied","Data":"ab92865792d0d73fc866116afcdc195135171eb9ba83169f72edbf36ea4c5920"}
Nov 02 15:47:17.301754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.301510    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab92865792d0d73fc866116afcdc195135171eb9ba83169f72edbf36ea4c5920"
Nov 02 15:47:17.311044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.311016    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:17.427532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.427496    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"]
Nov 02 15:47:17.427808 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.427782    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy" containerID="cri-o://dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f" gracePeriod=30
Nov 02 15:47:17.485385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.485332    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:17.821671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.821024    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:17.970871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:17.970831    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:18.122743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.122701    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"]
Nov 02 15:47:18.123492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.123469    7019 cpu_manager.go:414] "containerMap: removing container" podUID="501cd36a-77b8-429b-b253-949f798f9a9d" containerName="revoke-keys"
Nov 02 15:47:18.123492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.123492    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501cd36a-77b8-429b-b253-949f798f9a9d" containerName="revoke-keys"
Nov 02 15:47:18.123675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.123637    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="501cd36a-77b8-429b-b253-949f798f9a9d" containerName="revoke-keys"
Nov 02 15:47:18.127548 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.127527    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.145485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.145452    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"]
Nov 02 15:47:18.171134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.171098    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:47:18.171349 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.171274    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:47:18.256986 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.256928    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.257223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.257126    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.257359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.257340    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45bg\" (UniqueName: \"kubernetes.io/projected/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-kube-api-access-p45bg\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.358807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.358772    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45bg\" (UniqueName: \"kubernetes.io/projected/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-kube-api-access-p45bg\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.359000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.358869    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.359000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.358946    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.359676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.359638    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.361450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.361413    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.429380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.429258    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45bg\" (UniqueName: \"kubernetes.io/projected/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-kube-api-access-p45bg\") pod \"maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.442724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.442119    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:18.646354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.646322    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:18.884088 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:47:18.884056    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a1d9ed_7d2c_49e2_8cf8_a5456c3f4f1c.slice/crio-674334d8eb7d9f2564405ad33e600e505ec07614fc52883e0eea1c1c89c898c1 WatchSource:0}: Error finding container 674334d8eb7d9f2564405ad33e600e505ec07614fc52883e0eea1c1c89c898c1: Status 404 returned error can't find the container with id 674334d8eb7d9f2564405ad33e600e505ec07614fc52883e0eea1c1c89c898c1
Nov 02 15:47:18.884337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:18.884315    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"]
Nov 02 15:47:19.321362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:19.321319    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf" event={"ID":"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c","Type":"ContainerStarted","Data":"b7e90013e0bc53b822c9668e302ce7c6387c2567515d945f6eb6b55191faa04e"}
Nov 02 15:47:19.321545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:19.321385    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf" event={"ID":"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c","Type":"ContainerStarted","Data":"674334d8eb7d9f2564405ad33e600e505ec07614fc52883e0eea1c1c89c898c1"}
Nov 02 15:47:19.372736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:19.372680    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf" podStartSLOduration=1.372664562 podStartE2EDuration="1.372664562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:19.368629308 +0000 UTC m=+3033.622582933" watchObservedRunningTime="2026-11-02 15:47:19.372664562 +0000 UTC m=+3033.626618186"
Nov 02 15:47:20.325527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:20.325486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:20.332739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:20.332706    7019 generic.go:356] "Generic (PLEG): container finished" podID="c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" containerID="b7e90013e0bc53b822c9668e302ce7c6387c2567515d945f6eb6b55191faa04e" exitCode=0
Nov 02 15:47:20.332917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:20.332807    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf" event={"ID":"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c","Type":"ContainerDied","Data":"b7e90013e0bc53b822c9668e302ce7c6387c2567515d945f6eb6b55191faa04e"}
Nov 02 15:47:21.528499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.528466    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:21.619474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.619440    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-openshift-ca\") pod \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") "
Nov 02 15:47:21.619681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.619591    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-xks-ca\") pod \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") "
Nov 02 15:47:21.619777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.619704    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-kube-api-access-p45bg\" (UniqueName: \"kubernetes.io/projected/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-kube-api-access-p45bg\") pod \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\" (UID: \"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c\") "
Nov 02 15:47:21.619838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.619816    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-openshift-ca" pod "c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" (UID: "c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:21.620307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.620278    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:21.625066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.622256    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-xks-ca" pod "c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" (UID: "c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:21.625415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.625385    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-kube-api-access-p45bg" pod "c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" (UID: "c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c"). InnerVolumeSpecName "kube-api-access-p45bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:21.721846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.721804    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:21.721846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:21.721842    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p45bg\" (UniqueName: \"kubernetes.io/projected/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c-kube-api-access-p45bg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.023295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.022692    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:22.349048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.348967    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"
Nov 02 15:47:22.349048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.348998    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf" event={"ID":"c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c","Type":"ContainerDied","Data":"674334d8eb7d9f2564405ad33e600e505ec07614fc52883e0eea1c1c89c898c1"}
Nov 02 15:47:22.349048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.349048    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="674334d8eb7d9f2564405ad33e600e505ec07614fc52883e0eea1c1c89c898c1"
Nov 02 15:47:22.710951 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:22.710911    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:47:22.736445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.736422    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:47:22.836969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.836933    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf5e56a-3578-433f-9169-a1901aeb6de6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf5e56a-3578-433f-9169-a1901aeb6de6-istiod-ca-cert\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837015    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-envoy\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837046    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-data\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837117    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-credential-socket\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837211    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-socket\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837334    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf5e56a-3578-433f-9169-a1901aeb6de6-istiod-ca-cert" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:22.837489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837378    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-token\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-token\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837414    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-kube-api-access-7qt67\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-kube-api-access-7qt67\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837482    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-certs\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837510    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-podinfo\") pod \"4bf5e56a-3578-433f-9169-a1901aeb6de6\" (UID: \"4bf5e56a-3578-433f-9169-a1901aeb6de6\") "
Nov 02 15:47:22.837682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.837620    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-data" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:22.838141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.838108    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-credential-socket" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:22.838257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.838169    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bf5e56a-3578-433f-9169-a1901aeb6de6-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.838257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.838191    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.838455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.838431    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-certs" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:22.840549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.840516    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-kube-api-access-7qt67" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "kube-api-access-7qt67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:22.840703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.840680    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-envoy" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:22.840829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.840804    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-socket" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:22.841069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.841050    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-token" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:22.847508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.847480    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-podinfo" pod "4bf5e56a-3578-433f-9169-a1901aeb6de6" (UID: "4bf5e56a-3578-433f-9169-a1901aeb6de6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:47:22.897294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.897252    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"]
Nov 02 15:47:22.904690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.904638    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"]
Nov 02 15:47:22.939966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.939928    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.939966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.939969    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qt67\" (UniqueName: \"kubernetes.io/projected/4bf5e56a-3578-433f-9169-a1901aeb6de6-kube-api-access-7qt67\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.940187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.939986    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.940187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.940002    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.940187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.940015    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.940187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.940027    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:22.940187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:22.940039    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bf5e56a-3578-433f-9169-a1901aeb6de6-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:23.029066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.029017    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:23.358394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.358310    7019 generic.go:356] "Generic (PLEG): container finished" podID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerID="dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f" exitCode=0
Nov 02 15:47:23.358394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.358372    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"
Nov 02 15:47:23.358584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.358391    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" event={"ID":"4bf5e56a-3578-433f-9169-a1901aeb6de6","Type":"ContainerDied","Data":"dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f"}
Nov 02 15:47:23.358584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.358454    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99" event={"ID":"4bf5e56a-3578-433f-9169-a1901aeb6de6","Type":"ContainerDied","Data":"ff2cf1dd471cb9c035138c84bea299308aa13502269c4e3d2fd3388a57005ab0"}
Nov 02 15:47:23.358584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.358486    7019 scope.go:122] "RemoveContainer" containerID="dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f"
Nov 02 15:47:23.371074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.371047    7019 scope.go:122] "RemoveContainer" containerID="d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87"
Nov 02 15:47:23.381676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.381639    7019 scope.go:122] "RemoveContainer" containerID="dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f"
Nov 02 15:47:23.381994 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:23.381971    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f\": container with ID starting with dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f not found: ID does not exist" containerID="dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f"
Nov 02 15:47:23.382087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.382005    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f"} err="failed to get container status \"dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f\": rpc error: code = NotFound desc = could not find container \"dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f\": container with ID starting with dcf2661aa31815d8ef662552c18b73d4c7ade52f2b40d90b1e2b70821c91883f not found: ID does not exist"
Nov 02 15:47:23.382087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.382024    7019 scope.go:122] "RemoveContainer" containerID="d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87"
Nov 02 15:47:23.382365 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:23.382337    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87\": container with ID starting with d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87 not found: ID does not exist" containerID="d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87"
Nov 02 15:47:23.382430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.382377    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87"} err="failed to get container status \"d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87\": rpc error: code = NotFound desc = could not find container \"d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87\": container with ID starting with d716e9b01efcabb88bf8878ca6d8011a7bd60f54cb6238c4b78e7727865eec87 not found: ID does not exist"
Nov 02 15:47:23.428172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.428142    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"]
Nov 02 15:47:23.434277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.434238    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78d075-openshift-default-5c75868675-p4j99"]
Nov 02 15:47:23.706165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.706130    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:23.863411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:23.863291    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" path="/var/lib/kubelet/pods/4bf5e56a-3578-433f-9169-a1901aeb6de6/volumes"
Nov 02 15:47:24.238723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:24.238682    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"]
Nov 02 15:47:24.297734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:24.297703    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:25.368164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:25.368122    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:25.601256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:25.601210    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" podUID="d4209db8-105a-451a-9a43-719fe1a2a29b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:25.601463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:25.601345    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" podUID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:27.077295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:27.077258    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:28.651111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:28.651074    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"]
Nov 02 15:47:28.847554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:28.847507    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:29.010396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:29.009997    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:29.153973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:29.153934    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:30.372176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.372143    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"]
Nov 02 15:47:30.373129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.373105    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy"
Nov 02 15:47:30.373129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.373128    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy"
Nov 02 15:47:30.373340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.373158    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" containerName="revoke-keys"
Nov 02 15:47:30.373340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.373166    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" containerName="revoke-keys"
Nov 02 15:47:30.373454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.373387    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy"
Nov 02 15:47:30.373454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.373410    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy"
Nov 02 15:47:30.373454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.373424    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" containerName="revoke-keys"
Nov 02 15:47:30.378536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.378512    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.384675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.384632    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"]
Nov 02 15:47:30.385332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.385315    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy"
Nov 02 15:47:30.385411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.385333    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf5e56a-3578-433f-9169-a1901aeb6de6" containerName="istio-proxy"
Nov 02 15:47:30.389370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.389350    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.397949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.397921    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:47:30.398106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.398060    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:47:30.398212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.398187    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"]
Nov 02 15:47:30.418034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.417997    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"]
Nov 02 15:47:30.421619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.421587    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.421805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.421665    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zsc\" (UniqueName: \"kubernetes.io/projected/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-kube-api-access-k2zsc\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.421850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.421815    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.523556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.523516    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.523804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.523592    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zsc\" (UniqueName: \"kubernetes.io/projected/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-kube-api-access-k2zsc\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.523804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.523711    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.523804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.523763    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhp26\" (UniqueName: \"kubernetes.io/projected/e238251a-24ef-4390-8e3f-11ad8f0404f3-kube-api-access-vhp26\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.523953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.523850    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.524051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.524025    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.524281 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.524252    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.526135 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.526105    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.549822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.549785    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zsc\" (UniqueName: \"kubernetes.io/projected/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-kube-api-access-k2zsc\") pod \"maas-api-revoke-sub-f0f849b1-vkq2h\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.625688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.625568    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhp26\" (UniqueName: \"kubernetes.io/projected/e238251a-24ef-4390-8e3f-11ad8f0404f3-kube-api-access-vhp26\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.625688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.625673    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.625904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.625717    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.626345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.626325    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.628117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.628081    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.653008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.652974    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhp26\" (UniqueName: \"kubernetes.io/projected/e238251a-24ef-4390-8e3f-11ad8f0404f3-kube-api-access-vhp26\") pod \"maas-api-revoke-sub-96ff4281-c7jmw\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.697700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.697627    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:30.716532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.716506    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:30.812171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:30.806145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:31.013909 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:47:31.013876    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a30d0a_1fb9_4093_a0f5_bdbc50a357cb.slice/crio-d64079fc23b8d15a0c25f2c4c3ce0dfbe6c3bb1a58580c4c21c1866bbe739170 WatchSource:0}: Error finding container d64079fc23b8d15a0c25f2c4c3ce0dfbe6c3bb1a58580c4c21c1866bbe739170: Status 404 returned error can't find the container with id d64079fc23b8d15a0c25f2c4c3ce0dfbe6c3bb1a58580c4c21c1866bbe739170
Nov 02 15:47:31.015446 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:31.015417    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"]
Nov 02 15:47:31.093784 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:47:31.093754    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode238251a_24ef_4390_8e3f_11ad8f0404f3.slice/crio-3afd33c24986597ca1541012fc51e1a9ef919c70ccd4683ec5cb6ddcda7098fc WatchSource:0}: Error finding container 3afd33c24986597ca1541012fc51e1a9ef919c70ccd4683ec5cb6ddcda7098fc: Status 404 returned error can't find the container with id 3afd33c24986597ca1541012fc51e1a9ef919c70ccd4683ec5cb6ddcda7098fc
Nov 02 15:47:31.096123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:31.096092    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"]
Nov 02 15:47:31.525164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:31.525053    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h" event={"ID":"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb","Type":"ContainerStarted","Data":"6d6b87b8cc8c9cd152c2d9b0f0dd76fa98bc876ff65d9102eb0c5ec45b30e1c6"}
Nov 02 15:47:31.525164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:31.525116    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h" event={"ID":"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb","Type":"ContainerStarted","Data":"d64079fc23b8d15a0c25f2c4c3ce0dfbe6c3bb1a58580c4c21c1866bbe739170"}
Nov 02 15:47:31.526625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:31.526596    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw" event={"ID":"e238251a-24ef-4390-8e3f-11ad8f0404f3","Type":"ContainerStarted","Data":"3afd33c24986597ca1541012fc51e1a9ef919c70ccd4683ec5cb6ddcda7098fc"}
Nov 02 15:47:31.578168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:31.578103    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h" podStartSLOduration=1.5780836200000001 podStartE2EDuration="1.57808362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:31.575730618 +0000 UTC m=+3045.829684242" watchObservedRunningTime="2026-11-02 15:47:31.57808362 +0000 UTC m=+3045.832037245"
Nov 02 15:47:32.064154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.064112    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:32.254590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.254543    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:32.617662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.613807    7019 generic.go:356] "Generic (PLEG): container finished" podID="75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" containerID="6d6b87b8cc8c9cd152c2d9b0f0dd76fa98bc876ff65d9102eb0c5ec45b30e1c6" exitCode=0
Nov 02 15:47:32.617662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.614311    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h" event={"ID":"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb","Type":"ContainerDied","Data":"6d6b87b8cc8c9cd152c2d9b0f0dd76fa98bc876ff65d9102eb0c5ec45b30e1c6"}
Nov 02 15:47:32.617662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.617391    7019 generic.go:356] "Generic (PLEG): container finished" podID="e238251a-24ef-4390-8e3f-11ad8f0404f3" containerID="23eed0602d745c7570352b43ae31029daa8b2f864da7d17c6abdb662119189be" exitCode=0
Nov 02 15:47:32.617662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.617553    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw" event={"ID":"e238251a-24ef-4390-8e3f-11ad8f0404f3","Type":"ContainerDied","Data":"23eed0602d745c7570352b43ae31029daa8b2f864da7d17c6abdb662119189be"}
Nov 02 15:47:32.946074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.946037    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"]
Nov 02 15:47:32.946378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:32.946346    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf" podUID="4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" containerName="maas-api" containerID="cri-o://b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56" gracePeriod=30
Nov 02 15:47:33.044075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.044038    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:33.052067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.052037    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:47:33.057377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.057350    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:47:33.084336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.084298    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c214d04-6f64-4bea-b8e4-e98b175a8e44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c214d04-6f64-4bea-b8e4-e98b175a8e44-plugins-config-volume\") pod \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") "
Nov 02 15:47:33.084502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.084375    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c214d04-6f64-4bea-b8e4-e98b175a8e44-kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/8c214d04-6f64-4bea-b8e4-e98b175a8e44-kube-api-access-2l8sf\") pod \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\" (UID: \"8c214d04-6f64-4bea-b8e4-e98b175a8e44\") "
Nov 02 15:47:33.084502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.084473    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4209db8-105a-451a-9a43-719fe1a2a29b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4209db8-105a-451a-9a43-719fe1a2a29b-plugins-config-volume\") pod \"d4209db8-105a-451a-9a43-719fe1a2a29b\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") "
Nov 02 15:47:33.084658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.084552    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4209db8-105a-451a-9a43-719fe1a2a29b-kube-api-access-s7kff\" (UniqueName: \"kubernetes.io/projected/d4209db8-105a-451a-9a43-719fe1a2a29b-kube-api-access-s7kff\") pod \"d4209db8-105a-451a-9a43-719fe1a2a29b\" (UID: \"d4209db8-105a-451a-9a43-719fe1a2a29b\") "
Nov 02 15:47:33.086181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.085975    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4209db8-105a-451a-9a43-719fe1a2a29b-plugins-config-volume" pod "d4209db8-105a-451a-9a43-719fe1a2a29b" (UID: "d4209db8-105a-451a-9a43-719fe1a2a29b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:33.086181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.086146    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c214d04-6f64-4bea-b8e4-e98b175a8e44-plugins-config-volume" pod "8c214d04-6f64-4bea-b8e4-e98b175a8e44" (UID: "8c214d04-6f64-4bea-b8e4-e98b175a8e44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:33.089109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.089024    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4209db8-105a-451a-9a43-719fe1a2a29b-kube-api-access-s7kff" pod "d4209db8-105a-451a-9a43-719fe1a2a29b" (UID: "d4209db8-105a-451a-9a43-719fe1a2a29b"). InnerVolumeSpecName "kube-api-access-s7kff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:33.091000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.090966    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c214d04-6f64-4bea-b8e4-e98b175a8e44-kube-api-access-2l8sf" pod "8c214d04-6f64-4bea-b8e4-e98b175a8e44" (UID: "8c214d04-6f64-4bea-b8e4-e98b175a8e44"). InnerVolumeSpecName "kube-api-access-2l8sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:33.203076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.188071    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c214d04-6f64-4bea-b8e4-e98b175a8e44-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.203076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.199870    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l8sf\" (UniqueName: \"kubernetes.io/projected/8c214d04-6f64-4bea-b8e4-e98b175a8e44-kube-api-access-2l8sf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.203076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.199896    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4209db8-105a-451a-9a43-719fe1a2a29b-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.203076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.199911    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7kff\" (UniqueName: \"kubernetes.io/projected/d4209db8-105a-451a-9a43-719fe1a2a29b-kube-api-access-s7kff\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.518739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.518139    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:33.629192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.628829    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:47:33.630412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.630387    7019 generic.go:356] "Generic (PLEG): container finished" podID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" containerID="7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d" exitCode=0
Nov 02 15:47:33.630543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.630470    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" event={"ID":"8c214d04-6f64-4bea-b8e4-e98b175a8e44","Type":"ContainerDied","Data":"7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d"}
Nov 02 15:47:33.630543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.630509    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx" event={"ID":"8c214d04-6f64-4bea-b8e4-e98b175a8e44","Type":"ContainerDied","Data":"aadb4d4a34028fdf08c483e2d18d1cfdf97f6bce226d87541761ea8b58420d03"}
Nov 02 15:47:33.630543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.630533    7019 scope.go:122] "RemoveContainer" containerID="7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d"
Nov 02 15:47:33.630756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.630638    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"
Nov 02 15:47:33.635523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.635490    7019 generic.go:356] "Generic (PLEG): container finished" podID="d4209db8-105a-451a-9a43-719fe1a2a29b" containerID="17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a" exitCode=0
Nov 02 15:47:33.635712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.635581    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" event={"ID":"d4209db8-105a-451a-9a43-719fe1a2a29b","Type":"ContainerDied","Data":"17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a"}
Nov 02 15:47:33.635712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.635627    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25" event={"ID":"d4209db8-105a-451a-9a43-719fe1a2a29b","Type":"ContainerDied","Data":"dc2380d2d0b3e9ff148f2b61b026dc86f7eb027f04e11e56e65960c26ab53bfc"}
Nov 02 15:47:33.635850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.635743    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"
Nov 02 15:47:33.641333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.641100    7019 generic.go:356] "Generic (PLEG): container finished" podID="4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" containerID="b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56" exitCode=0
Nov 02 15:47:33.641501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.641480    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"
Nov 02 15:47:33.641660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.641626    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf" event={"ID":"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8","Type":"ContainerDied","Data":"b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56"}
Nov 02 15:47:33.641742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.641685    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf" event={"ID":"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8","Type":"ContainerDied","Data":"3d92efa6685b768092e2af8304ee3a2be8ccf5713ac6fb948814dc35eadad3a2"}
Nov 02 15:47:33.651481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.651405    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" podUID="729c2a0e-d79c-4f85-bd85-2375c58ce916" containerName="payload-processing" containerID="cri-o://19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182" gracePeriod=25
Nov 02 15:47:33.660058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.660033    7019 scope.go:122] "RemoveContainer" containerID="7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d"
Nov 02 15:47:33.660947 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:33.660390    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d\": container with ID starting with 7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d not found: ID does not exist" containerID="7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d"
Nov 02 15:47:33.660947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.660418    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d"} err="failed to get container status \"7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d\": rpc error: code = NotFound desc = could not find container \"7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d\": container with ID starting with 7231262f35e237fd7834567319e4c41f7699ad4cfd1a57f486c6fe277a8f1b8d not found: ID does not exist"
Nov 02 15:47:33.660947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.660434    7019 scope.go:122] "RemoveContainer" containerID="17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a"
Nov 02 15:47:33.753881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.753845    7019 scope.go:122] "RemoveContainer" containerID="17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a"
Nov 02 15:47:33.754509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.754483    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:33.761027 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:33.760838    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a\": container with ID starting with 17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a not found: ID does not exist" containerID="17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a"
Nov 02 15:47:33.761027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.760885    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a"} err="failed to get container status \"17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a\": rpc error: code = NotFound desc = could not find container \"17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a\": container with ID starting with 17ba02487a8a1b77cfd0ade34ab057706874570ae36ecaf9e115d9e13da5e01a not found: ID does not exist"
Nov 02 15:47:33.761027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.760912    7019 scope.go:122] "RemoveContainer" containerID="b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56"
Nov 02 15:47:33.775326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.774528    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-metrics-certs\") pod \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") "
Nov 02 15:47:33.775326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.774726    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-openshift-service-ca\") pod \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") "
Nov 02 15:47:33.775326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.774799    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-maas-api-tls\") pod \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") "
Nov 02 15:47:33.775326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.774836    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-kube-api-access-sn9rm\" (UniqueName: \"kubernetes.io/projected/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-kube-api-access-sn9rm\") pod \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\" (UID: \"4c6e5e8f-c6b3-4e83-a297-71954fa51dd8\") "
Nov 02 15:47:33.775686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.775453    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-openshift-service-ca" pod "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" (UID: "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:33.776079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.776051    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.786471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.786165    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-maas-api-tls" pod "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" (UID: "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:33.786471 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:33.786223    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4209db8_105a_451a_9a43_719fe1a2a29b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c214d04_6f64_4bea_b8e4_e98b175a8e44.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 15:47:33.787170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.787145    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-metrics-certs" pod "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" (UID: "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:33.808214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.803894    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-kube-api-access-sn9rm" pod "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" (UID: "4c6e5e8f-c6b3-4e83-a297-71954fa51dd8"). InnerVolumeSpecName "kube-api-access-sn9rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:33.810217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.810185    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"]
Nov 02 15:47:33.816124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.816097    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-78d075-54846d77ff-skn25"]
Nov 02 15:47:33.832815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.832798    7019 scope.go:122] "RemoveContainer" containerID="b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56"
Nov 02 15:47:33.833322 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:33.833298    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56\": container with ID starting with b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56 not found: ID does not exist" containerID="b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56"
Nov 02 15:47:33.833457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.833434    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56"} err="failed to get container status \"b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56\": rpc error: code = NotFound desc = could not find container \"b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56\": container with ID starting with b85a6f532524442d3bf2b8cec8d515d5af4f0bd53515fb6d377df6e91f642b56 not found: ID does not exist"
Nov 02 15:47:33.871249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.871213    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4209db8-105a-451a-9a43-719fe1a2a29b" path="/var/lib/kubelet/pods/d4209db8-105a-451a-9a43-719fe1a2a29b/volumes"
Nov 02 15:47:33.871889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.871865    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"]
Nov 02 15:47:33.872008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.871904    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-78d075-789b48f9c7-sqbjx"]
Nov 02 15:47:33.881902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.881872    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.881902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.881903    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.882092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.881921    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn9rm\" (UniqueName: \"kubernetes.io/projected/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8-kube-api-access-sn9rm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:33.962399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.962373    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:33.964770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:33.964746    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:34.001717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.001686    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"]
Nov 02 15:47:34.009614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.009577    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b955c5-55bd98b889-rwxbf"]
Nov 02 15:47:34.084906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.084813    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-xks-ca\") pod \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") "
Nov 02 15:47:34.084906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.084893    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-xks-ca\") pod \"e238251a-24ef-4390-8e3f-11ad8f0404f3\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") "
Nov 02 15:47:34.085139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.084933    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e238251a-24ef-4390-8e3f-11ad8f0404f3-kube-api-access-vhp26\" (UniqueName: \"kubernetes.io/projected/e238251a-24ef-4390-8e3f-11ad8f0404f3-kube-api-access-vhp26\") pod \"e238251a-24ef-4390-8e3f-11ad8f0404f3\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") "
Nov 02 15:47:34.085200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.085144    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-kube-api-access-k2zsc\" (UniqueName: \"kubernetes.io/projected/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-kube-api-access-k2zsc\") pod \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") "
Nov 02 15:47:34.085258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.085244    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-openshift-ca\") pod \"e238251a-24ef-4390-8e3f-11ad8f0404f3\" (UID: \"e238251a-24ef-4390-8e3f-11ad8f0404f3\") "
Nov 02 15:47:34.085371 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.085280    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-openshift-ca\") pod \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\" (UID: \"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb\") "
Nov 02 15:47:34.086301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.086261    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-openshift-ca" pod "e238251a-24ef-4390-8e3f-11ad8f0404f3" (UID: "e238251a-24ef-4390-8e3f-11ad8f0404f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:34.087189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.087155    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-openshift-ca" pod "75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" (UID: "75a30d0a-1fb9-4093-a0f5-bdbc50a357cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:34.088084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.088057    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-xks-ca" pod "75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" (UID: "75a30d0a-1fb9-4093-a0f5-bdbc50a357cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:34.090018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.089981    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-xks-ca" pod "e238251a-24ef-4390-8e3f-11ad8f0404f3" (UID: "e238251a-24ef-4390-8e3f-11ad8f0404f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:34.090459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.090435    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e238251a-24ef-4390-8e3f-11ad8f0404f3-kube-api-access-vhp26" pod "e238251a-24ef-4390-8e3f-11ad8f0404f3" (UID: "e238251a-24ef-4390-8e3f-11ad8f0404f3"). InnerVolumeSpecName "kube-api-access-vhp26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:34.090831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.090804    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-kube-api-access-k2zsc" pod "75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" (UID: "75a30d0a-1fb9-4093-a0f5-bdbc50a357cb"). InnerVolumeSpecName "kube-api-access-k2zsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:34.150564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.150526    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:34.187008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.186967    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:34.187008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.187002    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:34.187008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.187015    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhp26\" (UniqueName: \"kubernetes.io/projected/e238251a-24ef-4390-8e3f-11ad8f0404f3-kube-api-access-vhp26\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:34.187269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.187030    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2zsc\" (UniqueName: \"kubernetes.io/projected/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-kube-api-access-k2zsc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:34.187269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.187042    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e238251a-24ef-4390-8e3f-11ad8f0404f3-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:34.187269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.187054    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:34.654688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.654643    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"
Nov 02 15:47:34.655178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.654680    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h" event={"ID":"75a30d0a-1fb9-4093-a0f5-bdbc50a357cb","Type":"ContainerDied","Data":"d64079fc23b8d15a0c25f2c4c3ce0dfbe6c3bb1a58580c4c21c1866bbe739170"}
Nov 02 15:47:34.655178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.654737    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64079fc23b8d15a0c25f2c4c3ce0dfbe6c3bb1a58580c4c21c1866bbe739170"
Nov 02 15:47:34.663465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.663424    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw" event={"ID":"e238251a-24ef-4390-8e3f-11ad8f0404f3","Type":"ContainerDied","Data":"3afd33c24986597ca1541012fc51e1a9ef919c70ccd4683ec5cb6ddcda7098fc"}
Nov 02 15:47:34.663624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.663519    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3afd33c24986597ca1541012fc51e1a9ef919c70ccd4683ec5cb6ddcda7098fc"
Nov 02 15:47:34.663624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.663622    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"
Nov 02 15:47:34.759589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:34.759547    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" podUID="729c2a0e-d79c-4f85-bd85-2375c58ce916" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:35.382852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:35.382815    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:35.863163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:35.863127    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" path="/var/lib/kubelet/pods/4c6e5e8f-c6b3-4e83-a297-71954fa51dd8/volumes"
Nov 02 15:47:35.863790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:35.863770    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" path="/var/lib/kubelet/pods/8c214d04-6f64-4bea-b8e4-e98b175a8e44/volumes"
Nov 02 15:47:36.330643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:36.330562    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:36.512683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:36.512615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:37.075196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:37.075162    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:37.920283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:37.920244    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main" containerID="cri-o://2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14" gracePeriod=45
Nov 02 15:47:37.928157 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:37.928123    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" containerID="cri-o://853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8" gracePeriod=45
Nov 02 15:47:37.994847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:37.994813    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused"
Nov 02 15:47:38.233929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.233881    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:47:38.265086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.265057    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:47:38.350139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.349815    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b568cb-f879-4551-af12-9aecf54d8f6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b568cb-f879-4551-af12-9aecf54d8f6f-tls-certs\") pod \"d7b568cb-f879-4551-af12-9aecf54d8f6f\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") "
Nov 02 15:47:38.350139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.349935    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-model-cache\") pod \"d7b568cb-f879-4551-af12-9aecf54d8f6f\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") "
Nov 02 15:47:38.350388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.350361    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-model-cache" pod "d7b568cb-f879-4551-af12-9aecf54d8f6f" (UID: "d7b568cb-f879-4551-af12-9aecf54d8f6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.350672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.350622    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b568cb-f879-4551-af12-9aecf54d8f6f-kube-api-access-nz98q\" (UniqueName: \"kubernetes.io/projected/d7b568cb-f879-4551-af12-9aecf54d8f6f-kube-api-access-nz98q\") pod \"d7b568cb-f879-4551-af12-9aecf54d8f6f\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") "
Nov 02 15:47:38.350768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.350756    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-home\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-home\") pod \"d7b568cb-f879-4551-af12-9aecf54d8f6f\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") "
Nov 02 15:47:38.350911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.350894    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-dshm\") pod \"d7b568cb-f879-4551-af12-9aecf54d8f6f\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") "
Nov 02 15:47:38.351046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.351030    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-tmp-dir\") pod \"d7b568cb-f879-4551-af12-9aecf54d8f6f\" (UID: \"d7b568cb-f879-4551-af12-9aecf54d8f6f\") "
Nov 02 15:47:38.351368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.351213    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-tmp-dir" pod "d7b568cb-f879-4551-af12-9aecf54d8f6f" (UID: "d7b568cb-f879-4551-af12-9aecf54d8f6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.351368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.351343    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-home" pod "d7b568cb-f879-4551-af12-9aecf54d8f6f" (UID: "d7b568cb-f879-4551-af12-9aecf54d8f6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.351736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.351714    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.351831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.351739    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.351831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.351754    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.353462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.353436    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b568cb-f879-4551-af12-9aecf54d8f6f-kube-api-access-nz98q" pod "d7b568cb-f879-4551-af12-9aecf54d8f6f" (UID: "d7b568cb-f879-4551-af12-9aecf54d8f6f"). InnerVolumeSpecName "kube-api-access-nz98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:38.353730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.353710    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b568cb-f879-4551-af12-9aecf54d8f6f-tls-certs" pod "d7b568cb-f879-4551-af12-9aecf54d8f6f" (UID: "d7b568cb-f879-4551-af12-9aecf54d8f6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:38.354510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.354477    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-dshm" pod "d7b568cb-f879-4551-af12-9aecf54d8f6f" (UID: "d7b568cb-f879-4551-af12-9aecf54d8f6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.452878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.452845    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-tmp-dir\") pod \"60a729fd-8c15-46ab-81be-21f5052b2bce\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") "
Nov 02 15:47:38.452878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.452881    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-model-cache\") pod \"60a729fd-8c15-46ab-81be-21f5052b2bce\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") "
Nov 02 15:47:38.453139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.453006    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a729fd-8c15-46ab-81be-21f5052b2bce-tls-certs\" (UniqueName: \"kubernetes.io/secret/60a729fd-8c15-46ab-81be-21f5052b2bce-tls-certs\") pod \"60a729fd-8c15-46ab-81be-21f5052b2bce\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") "
Nov 02 15:47:38.453207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.453153    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-tmp-dir" pod "60a729fd-8c15-46ab-81be-21f5052b2bce" (UID: "60a729fd-8c15-46ab-81be-21f5052b2bce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.453271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.453162    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-model-cache" pod "60a729fd-8c15-46ab-81be-21f5052b2bce" (UID: "60a729fd-8c15-46ab-81be-21f5052b2bce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.453338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.453308    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a729fd-8c15-46ab-81be-21f5052b2bce-kube-api-access-l6prz\" (UniqueName: \"kubernetes.io/projected/60a729fd-8c15-46ab-81be-21f5052b2bce-kube-api-access-l6prz\") pod \"60a729fd-8c15-46ab-81be-21f5052b2bce\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") "
Nov 02 15:47:38.454054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.453700    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-home\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-home\") pod \"60a729fd-8c15-46ab-81be-21f5052b2bce\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") "
Nov 02 15:47:38.454054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.453917    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-dshm\") pod \"60a729fd-8c15-46ab-81be-21f5052b2bce\" (UID: \"60a729fd-8c15-46ab-81be-21f5052b2bce\") "
Nov 02 15:47:38.454689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.454660    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-home" pod "60a729fd-8c15-46ab-81be-21f5052b2bce" (UID: "60a729fd-8c15-46ab-81be-21f5052b2bce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.455750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.455717    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a729fd-8c15-46ab-81be-21f5052b2bce-tls-certs" pod "60a729fd-8c15-46ab-81be-21f5052b2bce" (UID: "60a729fd-8c15-46ab-81be-21f5052b2bce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:38.456097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.456068    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a729fd-8c15-46ab-81be-21f5052b2bce-kube-api-access-l6prz" pod "60a729fd-8c15-46ab-81be-21f5052b2bce" (UID: "60a729fd-8c15-46ab-81be-21f5052b2bce"). InnerVolumeSpecName "kube-api-access-l6prz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:38.457105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457075    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a729fd-8c15-46ab-81be-21f5052b2bce-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.457228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457109    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7b568cb-f879-4551-af12-9aecf54d8f6f-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.457228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457125    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6prz\" (UniqueName: \"kubernetes.io/projected/60a729fd-8c15-46ab-81be-21f5052b2bce-kube-api-access-l6prz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.457228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457141    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz98q\" (UniqueName: \"kubernetes.io/projected/d7b568cb-f879-4551-af12-9aecf54d8f6f-kube-api-access-nz98q\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.457228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457155    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.457228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457169    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.457228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457181    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.457228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.457193    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7b568cb-f879-4551-af12-9aecf54d8f6f-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.507775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.497837    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-dshm" pod "60a729fd-8c15-46ab-81be-21f5052b2bce" (UID: "60a729fd-8c15-46ab-81be-21f5052b2bce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:38.559021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.558935    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a729fd-8c15-46ab-81be-21f5052b2bce-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:38.736668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.735756    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:38.850751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.836383    7019 generic.go:356] "Generic (PLEG): container finished" podID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerID="2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14" exitCode=0
Nov 02 15:47:38.850751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.836527    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" event={"ID":"d7b568cb-f879-4551-af12-9aecf54d8f6f","Type":"ContainerDied","Data":"2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14"}
Nov 02 15:47:38.850751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.836566    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s" event={"ID":"d7b568cb-f879-4551-af12-9aecf54d8f6f","Type":"ContainerDied","Data":"22d558815b650de027bf6bf97795c9cc938ab72101c6c137c051c3b6cf01e11e"}
Nov 02 15:47:38.850751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.836590    7019 scope.go:122] "RemoveContainer" containerID="2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14"
Nov 02 15:47:38.850751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.836759    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"
Nov 02 15:47:38.880212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.877669    7019 generic.go:356] "Generic (PLEG): container finished" podID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerID="853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8" exitCode=0
Nov 02 15:47:38.880212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.877841    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" event={"ID":"60a729fd-8c15-46ab-81be-21f5052b2bce","Type":"ContainerDied","Data":"853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8"}
Nov 02 15:47:38.880212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.877898    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824" event={"ID":"60a729fd-8c15-46ab-81be-21f5052b2bce","Type":"ContainerDied","Data":"ae8d35bc68d2b857de71ba1e133cde47c65dc7930ee1d7f8a5b3d0d577944cec"}
Nov 02 15:47:38.880212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.877989    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"
Nov 02 15:47:38.894783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.894479    7019 scope.go:122] "RemoveContainer" containerID="2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14"
Nov 02 15:47:38.895394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.894983    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:38.895394 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:38.895016    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14\": container with ID starting with 2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14 not found: ID does not exist" containerID="2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14"
Nov 02 15:47:38.895394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.895059    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14"} err="failed to get container status \"2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14\": rpc error: code = NotFound desc = could not find container \"2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14\": container with ID starting with 2287597e0e808a4c1efc59ee41e9b0dc1b016409bb8bd929c9722cce83b69a14 not found: ID does not exist"
Nov 02 15:47:38.895394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.895083    7019 scope.go:122] "RemoveContainer" containerID="853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8"
Nov 02 15:47:38.908226 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.908183    7019 scope.go:122] "RemoveContainer" containerID="853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8"
Nov 02 15:47:38.908588 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:38.908562    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8\": container with ID starting with 853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8 not found: ID does not exist" containerID="853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8"
Nov 02 15:47:38.908730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.908598    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8"} err="failed to get container status \"853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8\": rpc error: code = NotFound desc = could not find container \"853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8\": container with ID starting with 853f2b8dbb0c3585b634de033aa521c4b92ebfea7109e99398a2c3e88caa1ee8 not found: ID does not exist"
Nov 02 15:47:38.924880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.924852    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"]
Nov 02 15:47:38.928283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.928259    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/premium-w4-7ecfe1-kserve-7bc6648fff-nw47s"]
Nov 02 15:47:38.988202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.988167    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"]
Nov 02 15:47:38.995137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:38.995108    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7ecfe1/simulator-w4-7ecfe1-kserve-5f7fdbbcb6-q7824"]
Nov 02 15:47:39.248091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.248057    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:39.260077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.260042    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2" podUID="3b712526-9385-4c26-b37e-8b0bee370de1" containerName="main" containerID="cri-o://97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052" gracePeriod=45
Nov 02 15:47:39.541028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.541000    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:47:39.699677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.698824    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-home\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-home\") pod \"3b712526-9385-4c26-b37e-8b0bee370de1\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") "
Nov 02 15:47:39.699677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.698885    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-dshm\") pod \"3b712526-9385-4c26-b37e-8b0bee370de1\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") "
Nov 02 15:47:39.699677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.698930    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-tmp-dir\") pod \"3b712526-9385-4c26-b37e-8b0bee370de1\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") "
Nov 02 15:47:39.699677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.698966    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-model-cache\") pod \"3b712526-9385-4c26-b37e-8b0bee370de1\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") "
Nov 02 15:47:39.699677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.699096    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b712526-9385-4c26-b37e-8b0bee370de1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b712526-9385-4c26-b37e-8b0bee370de1-tls-certs\") pod \"3b712526-9385-4c26-b37e-8b0bee370de1\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") "
Nov 02 15:47:39.699677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.699294    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b712526-9385-4c26-b37e-8b0bee370de1-kube-api-access-rwdv6\" (UniqueName: \"kubernetes.io/projected/3b712526-9385-4c26-b37e-8b0bee370de1-kube-api-access-rwdv6\") pod \"3b712526-9385-4c26-b37e-8b0bee370de1\" (UID: \"3b712526-9385-4c26-b37e-8b0bee370de1\") "
Nov 02 15:47:39.703742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.700835    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-tmp-dir" pod "3b712526-9385-4c26-b37e-8b0bee370de1" (UID: "3b712526-9385-4c26-b37e-8b0bee370de1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:39.703742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.701024    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-home" pod "3b712526-9385-4c26-b37e-8b0bee370de1" (UID: "3b712526-9385-4c26-b37e-8b0bee370de1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:39.703742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.701898    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-model-cache" pod "3b712526-9385-4c26-b37e-8b0bee370de1" (UID: "3b712526-9385-4c26-b37e-8b0bee370de1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:39.709729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.709068    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-dshm" pod "3b712526-9385-4c26-b37e-8b0bee370de1" (UID: "3b712526-9385-4c26-b37e-8b0bee370de1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:47:39.709729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.709433    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b712526-9385-4c26-b37e-8b0bee370de1-kube-api-access-rwdv6" pod "3b712526-9385-4c26-b37e-8b0bee370de1" (UID: "3b712526-9385-4c26-b37e-8b0bee370de1"). InnerVolumeSpecName "kube-api-access-rwdv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:39.715019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.714990    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b712526-9385-4c26-b37e-8b0bee370de1-tls-certs" pod "3b712526-9385-4c26-b37e-8b0bee370de1" (UID: "3b712526-9385-4c26-b37e-8b0bee370de1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:39.796720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.796625    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:39.801257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.801231    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b712526-9385-4c26-b37e-8b0bee370de1-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:39.801257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.801255    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdv6\" (UniqueName: \"kubernetes.io/projected/3b712526-9385-4c26-b37e-8b0bee370de1-kube-api-access-rwdv6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:39.801477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.801264    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:39.801477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.801275    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:39.801477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.801282    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:39.801477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.801290    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b712526-9385-4c26-b37e-8b0bee370de1-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:39.979949 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.979913    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:39.995725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.995670    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" path="/var/lib/kubelet/pods/60a729fd-8c15-46ab-81be-21f5052b2bce/volumes"
Nov 02 15:47:40.009671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:39.996610    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" path="/var/lib/kubelet/pods/d7b568cb-f879-4551-af12-9aecf54d8f6f/volumes"
Nov 02 15:47:40.043672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.042676    7019 generic.go:356] "Generic (PLEG): container finished" podID="3b712526-9385-4c26-b37e-8b0bee370de1" containerID="97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052" exitCode=0
Nov 02 15:47:40.043672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.042787    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2" event={"ID":"3b712526-9385-4c26-b37e-8b0bee370de1","Type":"ContainerDied","Data":"97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052"}
Nov 02 15:47:40.043672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.042827    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2" event={"ID":"3b712526-9385-4c26-b37e-8b0bee370de1","Type":"ContainerDied","Data":"27b5487d241afcd389b3f9b25c084e7ec65cbf3dc6f313ad4c304dae3100b313"}
Nov 02 15:47:40.043672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.042850    7019 scope.go:122] "RemoveContainer" containerID="97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052"
Nov 02 15:47:40.043672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.043005    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"
Nov 02 15:47:40.141678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.140943    7019 scope.go:122] "RemoveContainer" containerID="97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052"
Nov 02 15:47:40.141678 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:40.141339    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052\": container with ID starting with 97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052 not found: ID does not exist" containerID="97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052"
Nov 02 15:47:40.141678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.141371    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052"} err="failed to get container status \"97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052\": rpc error: code = NotFound desc = could not find container \"97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052\": container with ID starting with 97caec47b440fe9ff8bb99decadcb3d125a885fef778a1339f4910ee7f6f1052 not found: ID does not exist"
Nov 02 15:47:40.168580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.168546    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"]
Nov 02 15:47:40.174223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.174188    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f6f1a0/rate-test-model-f6f1a0-kserve-855549489c-8l2w2"]
Nov 02 15:47:40.616985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:40.616947    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:41.040139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:41.040049    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"]
Nov 02 15:47:41.043269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:41.043239    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce097e9-lrjgm"]
Nov 02 15:47:41.471421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:41.471382    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:41.623913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:41.623881    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:41.801201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:41.801107    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:41.906121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:41.906085    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b712526-9385-4c26-b37e-8b0bee370de1" path="/var/lib/kubelet/pods/3b712526-9385-4c26-b37e-8b0bee370de1/volumes"
Nov 02 15:47:41.907104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:41.907080    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b292c8dc-7d92-4272-bc56-bb2839596653" path="/var/lib/kubelet/pods/b292c8dc-7d92-4272-bc56-bb2839596653/volumes"
Nov 02 15:47:42.377594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.377555    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"]
Nov 02 15:47:42.378504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378481    7019 cpu_manager.go:414] "containerMap: removing container" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378505    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378528    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d4209db8-105a-451a-9a43-719fe1a2a29b" containerName="payload-pre-processing"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378536    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4209db8-105a-451a-9a43-719fe1a2a29b" containerName="payload-pre-processing"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378547    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3b712526-9385-4c26-b37e-8b0bee370de1" containerName="main"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378556    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b712526-9385-4c26-b37e-8b0bee370de1" containerName="main"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378580    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378588    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378600    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" containerName="maas-api"
Nov 02 15:47:42.378626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378607    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" containerName="maas-api"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378661    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e238251a-24ef-4390-8e3f-11ad8f0404f3" containerName="revoke-keys"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378669    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e238251a-24ef-4390-8e3f-11ad8f0404f3" containerName="revoke-keys"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378701    7019 cpu_manager.go:414] "containerMap: removing container" podUID="75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" containerName="revoke-keys"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378708    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" containerName="revoke-keys"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378735    7019 cpu_manager.go:414] "containerMap: removing container" podUID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" containerName="payload-processing"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378742    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" containerName="payload-processing"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378871    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a729fd-8c15-46ab-81be-21f5052b2bce" containerName="main"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378885    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4209db8-105a-451a-9a43-719fe1a2a29b" containerName="payload-pre-processing"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378894    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b712526-9385-4c26-b37e-8b0bee370de1" containerName="main"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378907    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b568cb-f879-4551-af12-9aecf54d8f6f" containerName="main"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378918    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6e5e8f-c6b3-4e83-a297-71954fa51dd8" containerName="maas-api"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378933    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e238251a-24ef-4390-8e3f-11ad8f0404f3" containerName="revoke-keys"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378945    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" containerName="revoke-keys"
Nov 02 15:47:42.379125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.378957    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c214d04-6f64-4bea-b8e4-e98b175a8e44" containerName="payload-processing"
Nov 02 15:47:42.384307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.384285    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.414172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.414139    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:42.417127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.417099    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"]
Nov 02 15:47:42.458202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.458158    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmbn8\" (UniqueName: \"kubernetes.io/projected/a1fb2374-8eae-454a-b3d5-bacdda81ba38-kube-api-access-wmbn8\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.462855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.462817    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.463035 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.462899    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.463575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.463554    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:47:42.470987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.470961    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:47:42.564839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.564803    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmbn8\" (UniqueName: \"kubernetes.io/projected/a1fb2374-8eae-454a-b3d5-bacdda81ba38-kube-api-access-wmbn8\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.565019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.564921    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.565019 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.564955    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.565643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.565613    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.567495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.567473    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.591975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.591941    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmbn8\" (UniqueName: \"kubernetes.io/projected/a1fb2374-8eae-454a-b3d5-bacdda81ba38-kube-api-access-wmbn8\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:42.698427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:42.698338    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:43.013748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:43.013703    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"]
Nov 02 15:47:43.015612 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:47:43.015584    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fb2374_8eae_454a_b3d5_bacdda81ba38.slice/crio-fe6170b76d53d81a00883588edc948e8893b9727b4c8825afea02a33fe5ea588 WatchSource:0}: Error finding container fe6170b76d53d81a00883588edc948e8893b9727b4c8825afea02a33fe5ea588: Status 404 returned error can't find the container with id fe6170b76d53d81a00883588edc948e8893b9727b4c8825afea02a33fe5ea588
Nov 02 15:47:43.103686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:43.103605    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9" event={"ID":"a1fb2374-8eae-454a-b3d5-bacdda81ba38","Type":"ContainerStarted","Data":"fe6170b76d53d81a00883588edc948e8893b9727b4c8825afea02a33fe5ea588"}
Nov 02 15:47:43.654727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:43.654597    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:43.796851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:43.796811    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:43.941469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:43.941380    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:44.110544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:44.110451    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:44.125204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:44.125163    7019 generic.go:356] "Generic (PLEG): container finished" podID="a1fb2374-8eae-454a-b3d5-bacdda81ba38" containerID="815b4a47af0b0433d7573e34a7f8f0493d5a51a6da536fa5c7808b48321275ea" exitCode=0
Nov 02 15:47:44.125415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:44.125354    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9" event={"ID":"a1fb2374-8eae-454a-b3d5-bacdda81ba38","Type":"ContainerDied","Data":"815b4a47af0b0433d7573e34a7f8f0493d5a51a6da536fa5c7808b48321275ea"}
Nov 02 15:47:44.259709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:44.259589    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:44.416098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:44.416065    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:44.759744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:44.759707    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" podUID="729c2a0e-d79c-4f85-bd85-2375c58ce916" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:47:45.338299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.338274    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:45.518206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.518110    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-xks-ca\") pod \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") "
Nov 02 15:47:45.518206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.518198    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-openshift-ca\") pod \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") "
Nov 02 15:47:45.518411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.518229    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fb2374-8eae-454a-b3d5-bacdda81ba38-kube-api-access-wmbn8\" (UniqueName: \"kubernetes.io/projected/a1fb2374-8eae-454a-b3d5-bacdda81ba38-kube-api-access-wmbn8\") pod \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\" (UID: \"a1fb2374-8eae-454a-b3d5-bacdda81ba38\") "
Nov 02 15:47:45.518574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.518546    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-openshift-ca" pod "a1fb2374-8eae-454a-b3d5-bacdda81ba38" (UID: "a1fb2374-8eae-454a-b3d5-bacdda81ba38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:45.518842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.518825    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:45.520455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.520422    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-xks-ca" pod "a1fb2374-8eae-454a-b3d5-bacdda81ba38" (UID: "a1fb2374-8eae-454a-b3d5-bacdda81ba38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:47:45.520559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.520465    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fb2374-8eae-454a-b3d5-bacdda81ba38-kube-api-access-wmbn8" pod "a1fb2374-8eae-454a-b3d5-bacdda81ba38" (UID: "a1fb2374-8eae-454a-b3d5-bacdda81ba38"). InnerVolumeSpecName "kube-api-access-wmbn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:45.620348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.620312    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1fb2374-8eae-454a-b3d5-bacdda81ba38-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:45.620348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.620345    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmbn8\" (UniqueName: \"kubernetes.io/projected/a1fb2374-8eae-454a-b3d5-bacdda81ba38-kube-api-access-wmbn8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:45.873085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:45.873052    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:46.069555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:46.069515    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:46.305825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:46.303932    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9" event={"ID":"a1fb2374-8eae-454a-b3d5-bacdda81ba38","Type":"ContainerDied","Data":"fe6170b76d53d81a00883588edc948e8893b9727b4c8825afea02a33fe5ea588"}
Nov 02 15:47:46.305825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:46.303997    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6170b76d53d81a00883588edc948e8893b9727b4c8825afea02a33fe5ea588"
Nov 02 15:47:46.305825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:46.304070    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"
Nov 02 15:47:46.820580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:46.820545    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:48.513735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:48.513695    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:50.080730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:50.080694    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:50.430979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:50.430944    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:50.650463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:50.650419    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:51.154929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:51.154811    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:51.329674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:51.322921    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:51.848671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:51.847675    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:52.034914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:52.034875    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"]
Nov 02 15:47:52.046996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:52.046961    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c25bf2-zbjjp"]
Nov 02 15:47:52.710721 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:52.710681    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:47:53.071034 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:53.070931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:53.209941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:53.209906    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:53.414268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:53.414231    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:53.879207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:53.879173    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f665d0-8c4f-449c-8c84-d9d88fa03830" path="/var/lib/kubelet/pods/f3f665d0-8c4f-449c-8c84-d9d88fa03830/volumes"
Nov 02 15:47:53.950350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:53.950280    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:47:54.052326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.052295    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729c2a0e-d79c-4f85-bd85-2375c58ce916-kube-api-access-2jvrg\" (UniqueName: \"kubernetes.io/projected/729c2a0e-d79c-4f85-bd85-2375c58ce916-kube-api-access-2jvrg\") pod \"729c2a0e-d79c-4f85-bd85-2375c58ce916\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") "
Nov 02 15:47:54.052530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.052388    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729c2a0e-d79c-4f85-bd85-2375c58ce916-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c2a0e-d79c-4f85-bd85-2375c58ce916-plugins-config-volume\") pod \"729c2a0e-d79c-4f85-bd85-2375c58ce916\" (UID: \"729c2a0e-d79c-4f85-bd85-2375c58ce916\") "
Nov 02 15:47:54.052871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.052846    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729c2a0e-d79c-4f85-bd85-2375c58ce916-plugins-config-volume" pod "729c2a0e-d79c-4f85-bd85-2375c58ce916" (UID: "729c2a0e-d79c-4f85-bd85-2375c58ce916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:47:54.053061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.053042    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/729c2a0e-d79c-4f85-bd85-2375c58ce916-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.054464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.054441    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729c2a0e-d79c-4f85-bd85-2375c58ce916-kube-api-access-2jvrg" pod "729c2a0e-d79c-4f85-bd85-2375c58ce916" (UID: "729c2a0e-d79c-4f85-bd85-2375c58ce916"). InnerVolumeSpecName "kube-api-access-2jvrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:47:54.154105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.154060    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvrg\" (UniqueName: \"kubernetes.io/projected/729c2a0e-d79c-4f85-bd85-2375c58ce916-kube-api-access-2jvrg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:47:54.395124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.395092    7019 generic.go:356] "Generic (PLEG): container finished" podID="729c2a0e-d79c-4f85-bd85-2375c58ce916" containerID="19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182" exitCode=0
Nov 02 15:47:54.395323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.395143    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"
Nov 02 15:47:54.395323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.395176    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" event={"ID":"729c2a0e-d79c-4f85-bd85-2375c58ce916","Type":"ContainerDied","Data":"19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182"}
Nov 02 15:47:54.395323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.395235    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb" event={"ID":"729c2a0e-d79c-4f85-bd85-2375c58ce916","Type":"ContainerDied","Data":"a4944fb8be5165b343f42b722ed9ffb91b3af257d51682a407310c77b3d3e4f3"}
Nov 02 15:47:54.395323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.395265    7019 scope.go:122] "RemoveContainer" containerID="19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182"
Nov 02 15:47:54.406954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.406934    7019 scope.go:122] "RemoveContainer" containerID="19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182"
Nov 02 15:47:54.407269 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:47:54.407247    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182\": container with ID starting with 19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182 not found: ID does not exist" containerID="19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182"
Nov 02 15:47:54.407380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.407276    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182"} err="failed to get container status \"19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182\": rpc error: code = NotFound desc = could not find container \"19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182\": container with ID starting with 19255c0030bdb798f9abb390e844cdddb1db5d1b136972f5219936829b3e2182 not found: ID does not exist"
Nov 02 15:47:54.455918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.455888    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"]
Nov 02 15:47:54.470106 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.470076    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-7ecfe1-f8484b6b5-pmzvb"]
Nov 02 15:47:54.762709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.762577    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:54.921724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:54.921666    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:55.319692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:55.317569    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:55.865026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:55.864993    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729c2a0e-d79c-4f85-bd85-2375c58ce916" path="/var/lib/kubelet/pods/729c2a0e-d79c-4f85-bd85-2375c58ce916/volumes"
Nov 02 15:47:56.857740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:56.857658    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:57.293420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:57.293383    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:57.448813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:57.448778    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:58.699545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:58.699502    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:59.199819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:59.199771    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:47:59.339910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:47:59.339879    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:00.393294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:00.393253    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:00.572909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:00.572873    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:00.754728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:00.747460    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:00.929727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:00.929683    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:01.657168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:01.657133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:02.034904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:02.034795    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:03.716327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:03.716240    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:05.387855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:05.387809    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:06.508544 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:06.508508    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:06.715513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:06.715422    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:07.733525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:07.733486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:08.902071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:08.902036    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:09.228354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:09.228269    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:09.410988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:09.410953    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:09.587294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:09.587202    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:11.392971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:11.392934    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:12.713473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:12.713433    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"]
Nov 02 15:48:12.713920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:12.713697    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6" podUID="10c98ff2-f44f-4453-bff2-eaa67af0869c" containerName="maas-api" containerID="cri-o://50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305" gracePeriod=30
Nov 02 15:48:12.819480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:12.819439    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:13.312087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.311295    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:48:13.369444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.368120    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:13.475139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.475047    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-maas-api-tls\") pod \"10c98ff2-f44f-4453-bff2-eaa67af0869c\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") "
Nov 02 15:48:13.475565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.475182    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-metrics-certs\") pod \"10c98ff2-f44f-4453-bff2-eaa67af0869c\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") "
Nov 02 15:48:13.475565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.475275    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c98ff2-f44f-4453-bff2-eaa67af0869c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c98ff2-f44f-4453-bff2-eaa67af0869c-openshift-service-ca\") pod \"10c98ff2-f44f-4453-bff2-eaa67af0869c\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") "
Nov 02 15:48:13.475565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.475509    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c98ff2-f44f-4453-bff2-eaa67af0869c-kube-api-access-gz9pr\" (UniqueName: \"kubernetes.io/projected/10c98ff2-f44f-4453-bff2-eaa67af0869c-kube-api-access-gz9pr\") pod \"10c98ff2-f44f-4453-bff2-eaa67af0869c\" (UID: \"10c98ff2-f44f-4453-bff2-eaa67af0869c\") "
Nov 02 15:48:13.475947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.475915    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c98ff2-f44f-4453-bff2-eaa67af0869c-openshift-service-ca" pod "10c98ff2-f44f-4453-bff2-eaa67af0869c" (UID: "10c98ff2-f44f-4453-bff2-eaa67af0869c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:13.476322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.476298    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10c98ff2-f44f-4453-bff2-eaa67af0869c-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:13.479105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.479080    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-metrics-certs" pod "10c98ff2-f44f-4453-bff2-eaa67af0869c" (UID: "10c98ff2-f44f-4453-bff2-eaa67af0869c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:13.479228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.479202    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c98ff2-f44f-4453-bff2-eaa67af0869c-kube-api-access-gz9pr" pod "10c98ff2-f44f-4453-bff2-eaa67af0869c" (UID: "10c98ff2-f44f-4453-bff2-eaa67af0869c"). InnerVolumeSpecName "kube-api-access-gz9pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:13.479228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.479228    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-maas-api-tls" pod "10c98ff2-f44f-4453-bff2-eaa67af0869c" (UID: "10c98ff2-f44f-4453-bff2-eaa67af0869c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:13.480063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.480037    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"]
Nov 02 15:48:13.483377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.483351    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-7ecfe1-4f4sf"]
Nov 02 15:48:13.566439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.554965    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.600366    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.600401    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10c98ff2-f44f-4453-bff2-eaa67af0869c-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.600417    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9pr\" (UniqueName: \"kubernetes.io/projected/10c98ff2-f44f-4453-bff2-eaa67af0869c-kube-api-access-gz9pr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.601554    7019 generic.go:356] "Generic (PLEG): container finished" podID="10c98ff2-f44f-4453-bff2-eaa67af0869c" containerID="50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305" exitCode=0
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.601604    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6" event={"ID":"10c98ff2-f44f-4453-bff2-eaa67af0869c","Type":"ContainerDied","Data":"50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305"}
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.601793    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6" event={"ID":"10c98ff2-f44f-4453-bff2-eaa67af0869c","Type":"ContainerDied","Data":"963af7d03c35d7c32e85dea66ddd2f7255c99bac2e7ae91eb52e997d471fc5e6"}
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.601887    7019 scope.go:122] "RemoveContainer" containerID="50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305"
Nov 02 15:48:13.604259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.602250    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"
Nov 02 15:48:13.618141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.618117    7019 scope.go:122] "RemoveContainer" containerID="50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305"
Nov 02 15:48:13.618513 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:48:13.618489    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305\": container with ID starting with 50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305 not found: ID does not exist" containerID="50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305"
Nov 02 15:48:13.618581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.618528    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305"} err="failed to get container status \"50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305\": rpc error: code = NotFound desc = could not find container \"50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305\": container with ID starting with 50c79d217c64db518e3f622862a2710ad879d04783bf57aa8b5b3e2d2a37f305 not found: ID does not exist"
Nov 02 15:48:13.658734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.658696    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"]
Nov 02 15:48:13.667014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.666988    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7ecfe1-67dbfc8f8b-9jxh6"]
Nov 02 15:48:13.700401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.700373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:13.863781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.863755    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c98ff2-f44f-4453-bff2-eaa67af0869c" path="/var/lib/kubelet/pods/10c98ff2-f44f-4453-bff2-eaa67af0869c/volumes"
Nov 02 15:48:13.864253 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.864239    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c" path="/var/lib/kubelet/pods/c1a1d9ed-7d2c-49e2-8cf8-a5456c3f4f1c/volumes"
Nov 02 15:48:13.943268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:13.943209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:14.106462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:14.106421    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:15.796049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:15.796001    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:16.796673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:16.795812    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:48:17.443677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:17.443618    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:19.159751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:19.159713    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:20.402208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:20.402168    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:20.546823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:20.546699    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:21.155981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:21.155953    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:21.292522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:21.292480    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:21.488849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:21.488751    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:22.707326 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:48:22.707275    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:48:23.182992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:23.182951    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:24.157976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.157942    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"]
Nov 02 15:48:24.158595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158579    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a1fb2374-8eae-454a-b3d5-bacdda81ba38" containerName="revoke-keys"
Nov 02 15:48:24.158643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158597    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1fb2374-8eae-454a-b3d5-bacdda81ba38" containerName="revoke-keys"
Nov 02 15:48:24.158643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158637    7019 cpu_manager.go:414] "containerMap: removing container" podUID="729c2a0e-d79c-4f85-bd85-2375c58ce916" containerName="payload-processing"
Nov 02 15:48:24.158643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158643    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729c2a0e-d79c-4f85-bd85-2375c58ce916" containerName="payload-processing"
Nov 02 15:48:24.158761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158667    7019 cpu_manager.go:414] "containerMap: removing container" podUID="10c98ff2-f44f-4453-bff2-eaa67af0869c" containerName="maas-api"
Nov 02 15:48:24.158761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158673    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c98ff2-f44f-4453-bff2-eaa67af0869c" containerName="maas-api"
Nov 02 15:48:24.158835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158765    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1fb2374-8eae-454a-b3d5-bacdda81ba38" containerName="revoke-keys"
Nov 02 15:48:24.158835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158781    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="729c2a0e-d79c-4f85-bd85-2375c58ce916" containerName="payload-processing"
Nov 02 15:48:24.158835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.158788    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c98ff2-f44f-4453-bff2-eaa67af0869c" containerName="maas-api"
Nov 02 15:48:24.162517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.162494    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.187533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.187504    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"]
Nov 02 15:48:24.203927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.203898    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:48:24.204263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.204247    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:48:24.317909 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.317868    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mj6\" (UniqueName: \"kubernetes.io/projected/826d955c-0781-4315-baeb-31660f76591f-kube-api-access-z9mj6\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.318122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.318005    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826d955c-0781-4315-baeb-31660f76591f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.318122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.318077    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826d955c-0781-4315-baeb-31660f76591f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.419843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.419768    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mj6\" (UniqueName: \"kubernetes.io/projected/826d955c-0781-4315-baeb-31660f76591f-kube-api-access-z9mj6\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.419990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.419869    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826d955c-0781-4315-baeb-31660f76591f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.419990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.419921    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826d955c-0781-4315-baeb-31660f76591f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.420700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.420679    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826d955c-0781-4315-baeb-31660f76591f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.422155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.422132    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826d955c-0781-4315-baeb-31660f76591f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.443849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.443823    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mj6\" (UniqueName: \"kubernetes.io/projected/826d955c-0781-4315-baeb-31660f76591f-kube-api-access-z9mj6\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.483290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.483261    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:24.790511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.790470    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"]
Nov 02 15:48:24.793497 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:48:24.793447    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826d955c_0781_4315_baeb_31660f76591f.slice/crio-10f2f1d4db489c03af9a80957748de5b248f8a6fcfee91f85677d230a803c627 WatchSource:0}: Error finding container 10f2f1d4db489c03af9a80957748de5b248f8a6fcfee91f85677d230a803c627: Status 404 returned error can't find the container with id 10f2f1d4db489c03af9a80957748de5b248f8a6fcfee91f85677d230a803c627
Nov 02 15:48:24.834894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.834863    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs" event={"ID":"826d955c-0781-4315-baeb-31660f76591f","Type":"ContainerStarted","Data":"10f2f1d4db489c03af9a80957748de5b248f8a6fcfee91f85677d230a803c627"}
Nov 02 15:48:24.853180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:24.853156    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:25.004263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:25.004125    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:25.216127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:25.216099    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:25.858490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:25.858449    7019 generic.go:356] "Generic (PLEG): container finished" podID="826d955c-0781-4315-baeb-31660f76591f" containerID="5bbf31be76d4c2d0ddfecedf9a25d4f4309497030895dcfd495a4dd57e072d7c" exitCode=0
Nov 02 15:48:25.864232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:25.864201    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs" event={"ID":"826d955c-0781-4315-baeb-31660f76591f","Type":"ContainerDied","Data":"5bbf31be76d4c2d0ddfecedf9a25d4f4309497030895dcfd495a4dd57e072d7c"}
Nov 02 15:48:26.803671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:26.803000    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:27.087239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.087215    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:27.162230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.162190    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826d955c-0781-4315-baeb-31660f76591f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826d955c-0781-4315-baeb-31660f76591f-maas-api-openshift-ca\") pod \"826d955c-0781-4315-baeb-31660f76591f\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") "
Nov 02 15:48:27.162401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.162325    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826d955c-0781-4315-baeb-31660f76591f-kube-api-access-z9mj6\" (UniqueName: \"kubernetes.io/projected/826d955c-0781-4315-baeb-31660f76591f-kube-api-access-z9mj6\") pod \"826d955c-0781-4315-baeb-31660f76591f\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") "
Nov 02 15:48:27.162401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.162379    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826d955c-0781-4315-baeb-31660f76591f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826d955c-0781-4315-baeb-31660f76591f-maas-api-xks-ca\") pod \"826d955c-0781-4315-baeb-31660f76591f\" (UID: \"826d955c-0781-4315-baeb-31660f76591f\") "
Nov 02 15:48:27.162675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.162627    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826d955c-0781-4315-baeb-31660f76591f-maas-api-openshift-ca" pod "826d955c-0781-4315-baeb-31660f76591f" (UID: "826d955c-0781-4315-baeb-31660f76591f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:27.163113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.163077    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/826d955c-0781-4315-baeb-31660f76591f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:27.164638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.164610    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826d955c-0781-4315-baeb-31660f76591f-kube-api-access-z9mj6" pod "826d955c-0781-4315-baeb-31660f76591f" (UID: "826d955c-0781-4315-baeb-31660f76591f"). InnerVolumeSpecName "kube-api-access-z9mj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:27.165741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.165717    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826d955c-0781-4315-baeb-31660f76591f-maas-api-xks-ca" pod "826d955c-0781-4315-baeb-31660f76591f" (UID: "826d955c-0781-4315-baeb-31660f76591f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:27.264865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.264824    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mj6\" (UniqueName: \"kubernetes.io/projected/826d955c-0781-4315-baeb-31660f76591f-kube-api-access-z9mj6\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:27.264865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.264864    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/826d955c-0781-4315-baeb-31660f76591f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:27.876501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.876471    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"
Nov 02 15:48:27.876501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.876490    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs" event={"ID":"826d955c-0781-4315-baeb-31660f76591f","Type":"ContainerDied","Data":"10f2f1d4db489c03af9a80957748de5b248f8a6fcfee91f85677d230a803c627"}
Nov 02 15:48:27.876925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:27.876536    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f2f1d4db489c03af9a80957748de5b248f8a6fcfee91f85677d230a803c627"
Nov 02 15:48:28.485212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:28.485173    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:30.377176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:30.377139    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:30.813290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:30.813198    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:30.909120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:30.909086    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"]
Nov 02 15:48:30.922373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:30.922343    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-b55da1-jtdj9"]
Nov 02 15:48:30.973726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:30.971519    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:31.448372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:31.448333    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:31.610772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:31.610737    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:31.862729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:31.862702    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1fb2374-8eae-454a-b3d5-bacdda81ba38" path="/var/lib/kubelet/pods/a1fb2374-8eae-454a-b3d5-bacdda81ba38/volumes"
Nov 02 15:48:32.685750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:32.685702    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:34.364520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:34.364484    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:36.039426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.039392    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:36.361892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.361857    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:36.765475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.765377    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"]
Nov 02 15:48:36.766258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.766234    7019 cpu_manager.go:414] "containerMap: removing container" podUID="826d955c-0781-4315-baeb-31660f76591f" containerName="revoke-keys"
Nov 02 15:48:36.766258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.766255    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826d955c-0781-4315-baeb-31660f76591f" containerName="revoke-keys"
Nov 02 15:48:36.766459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.766408    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="826d955c-0781-4315-baeb-31660f76591f" containerName="revoke-keys"
Nov 02 15:48:36.771189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.771167    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.808010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.806867    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8537e9-openshift-default-dockercfg-t8tnp\""
Nov 02 15:48:36.828081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.828045    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"]
Nov 02 15:48:36.862216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862186    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77765a7-6626-4005-b10c-db1068f65c21-istio-podinfo\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.862538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862517    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-certs\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.862685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862665    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-socket\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.862798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862702    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-data\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.862798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862786    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-istio-token\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.862887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862821    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-credential-socket\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.862932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862886    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffc8\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-kube-api-access-lffc8\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.862975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862959    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-envoy\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.863012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.862999    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77765a7-6626-4005-b10c-db1068f65c21-istiod-ca-cert\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.867566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.867537    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:36.972733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.972685    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-certs\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.972733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.972737    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-socket\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.973012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.972782    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-data\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.973012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.972868    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-istio-token\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.973012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.972914    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-credential-socket\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.973181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.973022    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffc8\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-kube-api-access-lffc8\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.973181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.973071    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-envoy\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.973181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.973156    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77765a7-6626-4005-b10c-db1068f65c21-istiod-ca-cert\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.973342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.973241    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77765a7-6626-4005-b10c-db1068f65c21-istio-podinfo\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.974471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.974438    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-credential-socket\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.978302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.975316    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-socket\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.978515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.975892    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-certs\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.978722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.976954    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77765a7-6626-4005-b10c-db1068f65c21-istiod-ca-cert\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.978920 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.977971    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-data\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:36.999570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:36.999532    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-envoy\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:37.044719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.031503    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffc8\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-kube-api-access-lffc8\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:37.072669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.037349    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77765a7-6626-4005-b10c-db1068f65c21-istio-podinfo\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:37.072938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.052420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:37.089683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.084662    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-istio-token\") pod \"e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") " pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:37.122668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.122535    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:37.248191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.247121    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:37.409864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.399398    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:37.763088 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:48:37.763021    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77765a7_6626_4005_b10c_db1068f65c21.slice/crio-90c3b255c92713703885a7ebea98d9eacb56d7c302226efcadec961e95d75daf WatchSource:0}: Error finding container 90c3b255c92713703885a7ebea98d9eacb56d7c302226efcadec961e95d75daf: Status 404 returned error can't find the container with id 90c3b255c92713703885a7ebea98d9eacb56d7c302226efcadec961e95d75daf
Nov 02 15:48:37.770362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.767307    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"]
Nov 02 15:48:37.770362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.768115    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:48:37.770362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.768440    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:48:37.770362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.769778    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:48:37.972131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.972092    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" event={"ID":"e77765a7-6626-4005-b10c-db1068f65c21","Type":"ContainerStarted","Data":"d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891"}
Nov 02 15:48:37.972334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:37.972160    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" event={"ID":"e77765a7-6626-4005-b10c-db1068f65c21","Type":"ContainerStarted","Data":"90c3b255c92713703885a7ebea98d9eacb56d7c302226efcadec961e95d75daf"}
Nov 02 15:48:38.031867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.031758    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podStartSLOduration=2.031737661 podStartE2EDuration="2.031737661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:38.030004 +0000 UTC m=+3112.283957650" watchObservedRunningTime="2026-11-02 15:48:38.031737661 +0000 UTC m=+3112.285691286"
Nov 02 15:48:38.123182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.123137    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:38.125360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.125330    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Nov 02 15:48:38.125513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.125389    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Nov 02 15:48:38.274669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.273505    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:38.449979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.441793    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:38.616899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.616318    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:38.779256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.779165    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:38.949479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:38.949436    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:39.114953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.114916    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"]
Nov 02 15:48:39.115791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.115749    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy" containerID="cri-o://702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab" gracePeriod=30
Nov 02 15:48:39.130667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.130250    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:39.149704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.149671    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:39.622183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.622147    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:39.779570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.779538    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"]
Nov 02 15:48:39.785201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.785167    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.808785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.808753    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"]
Nov 02 15:48:39.869187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.869152    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsgg8\" (UniqueName: \"kubernetes.io/projected/3b465de7-2a1a-41f3-9c49-18a12fe2c596-kube-api-access-rsgg8\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.869383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.869324    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.869479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.869458    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.872064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.872041    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:48:39.872342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.872294    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:48:39.970964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.970932    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsgg8\" (UniqueName: \"kubernetes.io/projected/3b465de7-2a1a-41f3-9c49-18a12fe2c596-kube-api-access-rsgg8\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.971177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.971023    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.971177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.971123    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.971901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.971873    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.973507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.973481    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:39.987498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.987473    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:39.988531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.988512    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:48:39.995816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:39.995789    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsgg8\" (UniqueName: \"kubernetes.io/projected/3b465de7-2a1a-41f3-9c49-18a12fe2c596-kube-api-access-rsgg8\") pod \"maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:40.102962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:40.102928    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:40.246727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:40.236113    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:40.481287 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:48:40.481249    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b465de7_2a1a_41f3_9c49_18a12fe2c596.slice/crio-541be15653667f116b741a57543363202da1658c66d3466f0948db2eacf2bb5a WatchSource:0}: Error finding container 541be15653667f116b741a57543363202da1658c66d3466f0948db2eacf2bb5a: Status 404 returned error can't find the container with id 541be15653667f116b741a57543363202da1658c66d3466f0948db2eacf2bb5a
Nov 02 15:48:40.482218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:40.482191    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"]
Nov 02 15:48:40.618839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:40.618799    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:40.778280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:40.778203    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:40.996464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:40.996415    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs" event={"ID":"3b465de7-2a1a-41f3-9c49-18a12fe2c596","Type":"ContainerStarted","Data":"c34f9261de8697f97e9fdf16b1b9fe38c9793629b892764d39ebae152a9c94b2"}
Nov 02 15:48:40.996637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:40.996526    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs" event={"ID":"3b465de7-2a1a-41f3-9c49-18a12fe2c596","Type":"ContainerStarted","Data":"541be15653667f116b741a57543363202da1658c66d3466f0948db2eacf2bb5a"}
Nov 02 15:48:42.011633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:42.011593    7019 generic.go:356] "Generic (PLEG): container finished" podID="3b465de7-2a1a-41f3-9c49-18a12fe2c596" containerID="c34f9261de8697f97e9fdf16b1b9fe38c9793629b892764d39ebae152a9c94b2" exitCode=0
Nov 02 15:48:42.012099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:42.011715    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs" event={"ID":"3b465de7-2a1a-41f3-9c49-18a12fe2c596","Type":"ContainerDied","Data":"c34f9261de8697f97e9fdf16b1b9fe38c9793629b892764d39ebae152a9c94b2"}
Nov 02 15:48:42.133121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:42.132997    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:42.391364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:42.391319    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:42.562141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:42.562104    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:43.181827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.181798    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:43.229798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.229769    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-xks-ca\") pod \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") "
Nov 02 15:48:43.229966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.229947    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b465de7-2a1a-41f3-9c49-18a12fe2c596-kube-api-access-rsgg8\" (UniqueName: \"kubernetes.io/projected/3b465de7-2a1a-41f3-9c49-18a12fe2c596-kube-api-access-rsgg8\") pod \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") "
Nov 02 15:48:43.230170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.230153    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-openshift-ca\") pod \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\" (UID: \"3b465de7-2a1a-41f3-9c49-18a12fe2c596\") "
Nov 02 15:48:43.230588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.230557    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-openshift-ca" pod "3b465de7-2a1a-41f3-9c49-18a12fe2c596" (UID: "3b465de7-2a1a-41f3-9c49-18a12fe2c596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:43.230797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.230768    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:43.232451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.232423    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-xks-ca" pod "3b465de7-2a1a-41f3-9c49-18a12fe2c596" (UID: "3b465de7-2a1a-41f3-9c49-18a12fe2c596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:43.232642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.232627    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b465de7-2a1a-41f3-9c49-18a12fe2c596-kube-api-access-rsgg8" pod "3b465de7-2a1a-41f3-9c49-18a12fe2c596" (UID: "3b465de7-2a1a-41f3-9c49-18a12fe2c596"). InnerVolumeSpecName "kube-api-access-rsgg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:43.332418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.332342    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsgg8\" (UniqueName: \"kubernetes.io/projected/3b465de7-2a1a-41f3-9c49-18a12fe2c596-kube-api-access-rsgg8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:43.332418 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.332372    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b465de7-2a1a-41f3-9c49-18a12fe2c596-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:43.786149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:43.786110    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:44.030086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.030048    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"
Nov 02 15:48:44.030264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.030091    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs" event={"ID":"3b465de7-2a1a-41f3-9c49-18a12fe2c596","Type":"ContainerDied","Data":"541be15653667f116b741a57543363202da1658c66d3466f0948db2eacf2bb5a"}
Nov 02 15:48:44.030264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.030152    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541be15653667f116b741a57543363202da1658c66d3466f0948db2eacf2bb5a"
Nov 02 15:48:44.510863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.510755    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"]
Nov 02 15:48:44.519210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.519180    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"]
Nov 02 15:48:44.630767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.630732    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:44.832688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.832659    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:48:44.852509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852479    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-token\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-token\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.852675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852540    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-credential-socket\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.852675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852581    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istiod-ca-cert\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.852675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852624    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-podinfo\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.852830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852735    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-socket\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.852830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852800    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-certs\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.852944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852876    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-envoy\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.853009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852937    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-credential-socket" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.853009 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.852980    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-data\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.853126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853046    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istiod-ca-cert" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:44.853126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853061    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-kube-api-access-m8gm8\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-kube-api-access-m8gm8\") pod \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\" (UID: \"cef64b9b-b435-4fde-9efa-a3e7429dd1f2\") "
Nov 02 15:48:44.853247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853178    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-certs" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.853338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853316    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-data" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.853407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853335    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-socket" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.853761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853729    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.853761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853758    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.853946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853771    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.853946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853786    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.853946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.853795    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.855289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.855259    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-token" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:44.855414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.855348    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-envoy" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:48:44.855484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.855442    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-podinfo" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:48:44.855598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.855575    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-kube-api-access-m8gm8" pod "cef64b9b-b435-4fde-9efa-a3e7429dd1f2" (UID: "cef64b9b-b435-4fde-9efa-a3e7429dd1f2"). InnerVolumeSpecName "kube-api-access-m8gm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:44.955187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.955147    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.955187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.955180    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.955187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.955190    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8gm8\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-kube-api-access-m8gm8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:44.955457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:44.955199    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cef64b9b-b435-4fde-9efa-a3e7429dd1f2-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:45.039271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.039236    7019 generic.go:356] "Generic (PLEG): container finished" podID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerID="702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab" exitCode=0
Nov 02 15:48:45.039441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.039286    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"
Nov 02 15:48:45.039441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.039324    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" event={"ID":"cef64b9b-b435-4fde-9efa-a3e7429dd1f2","Type":"ContainerDied","Data":"702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab"}
Nov 02 15:48:45.039441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.039389    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5" event={"ID":"cef64b9b-b435-4fde-9efa-a3e7429dd1f2","Type":"ContainerDied","Data":"e717cf4ed0f5e0299c2543088a49daa47d8a818538ff3ae1af6f441f25d23775"}
Nov 02 15:48:45.039441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.039415    7019 scope.go:122] "RemoveContainer" containerID="702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab"
Nov 02 15:48:45.051514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.051494    7019 scope.go:122] "RemoveContainer" containerID="702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab"
Nov 02 15:48:45.051826 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:48:45.051806    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab\": container with ID starting with 702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab not found: ID does not exist" containerID="702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab"
Nov 02 15:48:45.051893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.051835    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab"} err="failed to get container status \"702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab\": rpc error: code = NotFound desc = could not find container \"702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab\": container with ID starting with 702b7f365bd082d80aa20a27b468352b78b800c6158228eab510252b7be34cab not found: ID does not exist"
Nov 02 15:48:45.104702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.104588    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"]
Nov 02 15:48:45.107928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.107905    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-b55da1-openshift-default-787ff9f87c-t4dv5"]
Nov 02 15:48:45.478444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.478414    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:45.862474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:45.862445    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" path="/var/lib/kubelet/pods/cef64b9b-b435-4fde-9efa-a3e7429dd1f2/volumes"
Nov 02 15:48:47.135476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:47.135433    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:48.796458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:48.796408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:50.186857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:50.186816    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:50.335267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:50.335225    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:50.379384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:50.379341    7019 scope.go:122] "RemoveContainer" containerID="2ea697898968508cd08d399ba43583efc9e49a0ec886a758f5e81feac4e675c6"
Nov 02 15:48:50.390048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:50.390014    7019 scope.go:122] "RemoveContainer" containerID="01352cf4f815564b72a10785cc594795a32569e0925782dd3e68d0b9ca16e5e6"
Nov 02 15:48:50.528485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:50.528394    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:52.687520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:52.687426    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:52.755894 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:48:52.755855    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:48:53.099059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.098966    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:53.470312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.470267    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"]
Nov 02 15:48:53.471292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.471264    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3b465de7-2a1a-41f3-9c49-18a12fe2c596" containerName="revoke-keys"
Nov 02 15:48:53.471292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.471291    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b465de7-2a1a-41f3-9c49-18a12fe2c596" containerName="revoke-keys"
Nov 02 15:48:53.471586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.471310    7019 cpu_manager.go:414] "containerMap: removing container" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy"
Nov 02 15:48:53.471586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.471318    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy"
Nov 02 15:48:53.471586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.471544    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b465de7-2a1a-41f3-9c49-18a12fe2c596" containerName="revoke-keys"
Nov 02 15:48:53.471586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.471557    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cef64b9b-b435-4fde-9efa-a3e7429dd1f2" containerName="istio-proxy"
Nov 02 15:48:53.477615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.477590    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.509708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.496139    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-af33a6-openshift-default-dockercfg-9jk49\""
Nov 02 15:48:53.509708 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.501878    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"]
Nov 02 15:48:53.556834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.556790    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:53.627675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.627539    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnnq\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-kube-api-access-7fnnq\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.627675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.627613    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istiod-ca-cert\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.627955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.627769    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-certs\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.627955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.627928    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.628084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.627972    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-token\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.628084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.628011    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-credential-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.628199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.628089    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-data\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.628199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.628112    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-podinfo\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.628199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.628148    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-envoy\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.716138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.715817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:53.739413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.739309    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-data\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.739688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.739663    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-podinfo\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.739829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.739814    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-envoy\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.740016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.739998    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnnq\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-kube-api-access-7fnnq\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.740148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.740134    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istiod-ca-cert\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.740306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.740292    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-certs\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.740554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.740538    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.740687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.740673    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-token\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.740824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.740810    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-credential-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.741527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.741505    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-credential-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.741938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.741919    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-data\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.743284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.743004    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-socket\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.743284 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.743190    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-certs\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.767671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.750588    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-podinfo\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.767951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.758531    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istiod-ca-cert\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.768088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.760869    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-envoy\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.788712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.779345    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-token\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.788712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.781296    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnnq\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-kube-api-access-7fnnq\") pod \"e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") " pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:53.811162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:53.811129    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:54.212136 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:48:54.212097    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b3fdc1_62fc_438e_aebe_c77c3b960f10.slice/crio-a84d16a7d842c8c3c2ada785870be5b8d11400903a9bac1e6e0088a2fbeae88b WatchSource:0}: Error finding container a84d16a7d842c8c3c2ada785870be5b8d11400903a9bac1e6e0088a2fbeae88b: Status 404 returned error can't find the container with id a84d16a7d842c8c3c2ada785870be5b8d11400903a9bac1e6e0088a2fbeae88b
Nov 02 15:48:54.212336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:54.212317    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"]
Nov 02 15:48:54.216062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:54.216009    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:48:54.216161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:54.216110    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:48:54.216161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:54.216154    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:48:54.290582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:54.290546    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:55.131836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.131798    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" event={"ID":"47b3fdc1-62fc-438e-aebe-c77c3b960f10","Type":"ContainerStarted","Data":"f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b"}
Nov 02 15:48:55.132250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.131866    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" event={"ID":"47b3fdc1-62fc-438e-aebe-c77c3b960f10","Type":"ContainerStarted","Data":"a84d16a7d842c8c3c2ada785870be5b8d11400903a9bac1e6e0088a2fbeae88b"}
Nov 02 15:48:55.212222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.212165    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podStartSLOduration=2.212150681 podStartE2EDuration="2.212150681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:55.195092665 +0000 UTC m=+3129.449046290" watchObservedRunningTime="2026-11-02 15:48:55.212150681 +0000 UTC m=+3129.466104306"
Nov 02 15:48:55.554015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.553932    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"]
Nov 02 15:48:55.558411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.558388    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.572255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.572227    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"]
Nov 02 15:48:55.581916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.581884    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:48:55.582103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.582077    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:48:55.582470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.582450    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.584715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.583482    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"]
Nov 02 15:48:55.622512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.622481    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"]
Nov 02 15:48:55.683718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.683677    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.683888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.683740    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.683888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.683833    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.683888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.683874    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.683993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.683978    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmblz\" (UniqueName: \"kubernetes.io/projected/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-kube-api-access-rmblz\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.684032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.684020    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chpp8\" (UniqueName: \"kubernetes.io/projected/be94e6d6-dfc6-4cdb-9178-55cab00910e6-kube-api-access-chpp8\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.785783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.785748    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.785980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.785793    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.785980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.785815    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.785980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.785844    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.785980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.785875    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmblz\" (UniqueName: \"kubernetes.io/projected/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-kube-api-access-rmblz\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.785980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.785911    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chpp8\" (UniqueName: \"kubernetes.io/projected/be94e6d6-dfc6-4cdb-9178-55cab00910e6-kube-api-access-chpp8\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.786642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.786616    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.786806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.786672    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.788513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.788492    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.788852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.788833    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.811745 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.811675    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:55.812210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.812182    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chpp8\" (UniqueName: \"kubernetes.io/projected/be94e6d6-dfc6-4cdb-9178-55cab00910e6-kube-api-access-chpp8\") pod \"maas-api-revoke-sub-6f46db37-vlnpr\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.812210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.812204    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmblz\" (UniqueName: \"kubernetes.io/projected/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-kube-api-access-rmblz\") pod \"maas-api-revoke-sub-beba5bb9-xh5q9\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.817990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.817970    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:55.884341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.884301    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:55.933469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.933114    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:55.935070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:55.934958    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:56.140172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:56.140131    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:56.141567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:56.141540    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:48:56.243090 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:48:56.243061    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5375aab_a4a3_4285_b6a6_e0eeb3123a3b.slice/crio-71c69e6697110d356bab0d58f2ae7586371e4a7d6abafa2127c34b5ef30a0d9e WatchSource:0}: Error finding container 71c69e6697110d356bab0d58f2ae7586371e4a7d6abafa2127c34b5ef30a0d9e: Status 404 returned error can't find the container with id 71c69e6697110d356bab0d58f2ae7586371e4a7d6abafa2127c34b5ef30a0d9e
Nov 02 15:48:56.243525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:56.243501    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"]
Nov 02 15:48:56.397670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:56.388154    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"]
Nov 02 15:48:56.424421 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:48:56.424376    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe94e6d6_dfc6_4cdb_9178_55cab00910e6.slice/crio-58a9afd9339af0385d943be66027052404a548b5823c95d98bb32b71e322aea1 WatchSource:0}: Error finding container 58a9afd9339af0385d943be66027052404a548b5823c95d98bb32b71e322aea1: Status 404 returned error can't find the container with id 58a9afd9339af0385d943be66027052404a548b5823c95d98bb32b71e322aea1
Nov 02 15:48:57.148827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.148776    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9" event={"ID":"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b","Type":"ContainerStarted","Data":"5f896f35117cb7db74d850a41b52f0595b6426b3aee997fcfbba4eafa305f5ba"}
Nov 02 15:48:57.149277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.148851    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9" event={"ID":"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b","Type":"ContainerStarted","Data":"71c69e6697110d356bab0d58f2ae7586371e4a7d6abafa2127c34b5ef30a0d9e"}
Nov 02 15:48:57.150609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.150582    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr" event={"ID":"be94e6d6-dfc6-4cdb-9178-55cab00910e6","Type":"ContainerStarted","Data":"f73369b86d50ebee2b120626e21804b579fa2f6da28b2640820b729036974d75"}
Nov 02 15:48:57.150761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.150628    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr" event={"ID":"be94e6d6-dfc6-4cdb-9178-55cab00910e6","Type":"ContainerStarted","Data":"58a9afd9339af0385d943be66027052404a548b5823c95d98bb32b71e322aea1"}
Nov 02 15:48:57.194693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.194623    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9" podStartSLOduration=2.19460991 podStartE2EDuration="2.19460991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:57.193444312 +0000 UTC m=+3131.447397935" watchObservedRunningTime="2026-11-02 15:48:57.19460991 +0000 UTC m=+3131.448563570"
Nov 02 15:48:57.237555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.237505    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr" podStartSLOduration=2.237491156 podStartE2EDuration="2.237491156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:57.237254309 +0000 UTC m=+3131.491207934" watchObservedRunningTime="2026-11-02 15:48:57.237491156 +0000 UTC m=+3131.491444807"
Nov 02 15:48:57.562333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.562296    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:57.735731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:57.731802    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:58.160854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:58.160812    7019 generic.go:356] "Generic (PLEG): container finished" podID="be94e6d6-dfc6-4cdb-9178-55cab00910e6" containerID="f73369b86d50ebee2b120626e21804b579fa2f6da28b2640820b729036974d75" exitCode=0
Nov 02 15:48:58.161376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:58.160904    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr" event={"ID":"be94e6d6-dfc6-4cdb-9178-55cab00910e6","Type":"ContainerDied","Data":"f73369b86d50ebee2b120626e21804b579fa2f6da28b2640820b729036974d75"}
Nov 02 15:48:58.164793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:58.164767    7019 generic.go:356] "Generic (PLEG): container finished" podID="b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" containerID="5f896f35117cb7db74d850a41b52f0595b6426b3aee997fcfbba4eafa305f5ba" exitCode=0
Nov 02 15:48:58.164897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:58.164849    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9" event={"ID":"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b","Type":"ContainerDied","Data":"5f896f35117cb7db74d850a41b52f0595b6426b3aee997fcfbba4eafa305f5ba"}
Nov 02 15:48:58.243697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:58.243633    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:59.351115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.350833    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:59.395631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.395149    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:48:59.462166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.462089    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:48:59.464876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.464850    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-openshift-ca\") pod \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") "
Nov 02 15:48:59.465257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.465233    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-xks-ca\") pod \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") "
Nov 02 15:48:59.465425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.465409    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-kube-api-access-rmblz\" (UniqueName: \"kubernetes.io/projected/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-kube-api-access-rmblz\") pod \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\" (UID: \"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b\") "
Nov 02 15:48:59.465608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.465579    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-openshift-ca" pod "b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" (UID: "b5375aab-a4a3-4285-b6a6-e0eeb3123a3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:59.466476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.466455    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.468367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.468338    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-xks-ca" pod "b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" (UID: "b5375aab-a4a3-4285-b6a6-e0eeb3123a3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:59.468936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.468910    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-kube-api-access-rmblz" pod "b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" (UID: "b5375aab-a4a3-4285-b6a6-e0eeb3123a3b"). InnerVolumeSpecName "kube-api-access-rmblz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:59.515167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.515130    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:48:59.535921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.535869    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" containerID="cri-o://c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63" gracePeriod=45
Nov 02 15:48:59.540992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.540920    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" containerID="cri-o://65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae" gracePeriod=45
Nov 02 15:48:59.573404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.567138    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-openshift-ca\") pod \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") "
Nov 02 15:48:59.573404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.567245    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-xks-ca\") pod \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") "
Nov 02 15:48:59.573404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.567446    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be94e6d6-dfc6-4cdb-9178-55cab00910e6-kube-api-access-chpp8\" (UniqueName: \"kubernetes.io/projected/be94e6d6-dfc6-4cdb-9178-55cab00910e6-kube-api-access-chpp8\") pod \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\" (UID: \"be94e6d6-dfc6-4cdb-9178-55cab00910e6\") "
Nov 02 15:48:59.573404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.568564    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.573404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.568592    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmblz\" (UniqueName: \"kubernetes.io/projected/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b-kube-api-access-rmblz\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.573404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.569675    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-openshift-ca" pod "be94e6d6-dfc6-4cdb-9178-55cab00910e6" (UID: "be94e6d6-dfc6-4cdb-9178-55cab00910e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:48:59.573404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.572576    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be94e6d6-dfc6-4cdb-9178-55cab00910e6-kube-api-access-chpp8" pod "be94e6d6-dfc6-4cdb-9178-55cab00910e6" (UID: "be94e6d6-dfc6-4cdb-9178-55cab00910e6"). InnerVolumeSpecName "kube-api-access-chpp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:48:59.583150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.578119    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-xks-ca" pod "be94e6d6-dfc6-4cdb-9178-55cab00910e6" (UID: "be94e6d6-dfc6-4cdb-9178-55cab00910e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:48:59.601323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.584059    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused"
Nov 02 15:48:59.687597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.687507    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chpp8\" (UniqueName: \"kubernetes.io/projected/be94e6d6-dfc6-4cdb-9178-55cab00910e6-kube-api-access-chpp8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.687597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.687543    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.687597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.687558    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be94e6d6-dfc6-4cdb-9178-55cab00910e6-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:48:59.840677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:48:59.835250    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused"
Nov 02 15:49:00.034779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.034702    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:49:00.061707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.061583    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:00.145737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.145705    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:49:00.146901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.146872    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-dshm\") pod \"7f649146-86f7-45f2-b873-afc104a64c1c\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") "
Nov 02 15:49:00.147075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.147058    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-home\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-home\") pod \"7f649146-86f7-45f2-b873-afc104a64c1c\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") "
Nov 02 15:49:00.147198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.147180    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f649146-86f7-45f2-b873-afc104a64c1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f649146-86f7-45f2-b873-afc104a64c1c-tls-certs\") pod \"7f649146-86f7-45f2-b873-afc104a64c1c\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") "
Nov 02 15:49:00.147283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.147218    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-tmp-dir\") pod \"7f649146-86f7-45f2-b873-afc104a64c1c\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") "
Nov 02 15:49:00.147283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.147267    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f649146-86f7-45f2-b873-afc104a64c1c-kube-api-access-t4gf9\" (UniqueName: \"kubernetes.io/projected/7f649146-86f7-45f2-b873-afc104a64c1c-kube-api-access-t4gf9\") pod \"7f649146-86f7-45f2-b873-afc104a64c1c\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") "
Nov 02 15:49:00.147455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.147346    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-model-cache\") pod \"7f649146-86f7-45f2-b873-afc104a64c1c\" (UID: \"7f649146-86f7-45f2-b873-afc104a64c1c\") "
Nov 02 15:49:00.147512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.147462    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-home" pod "7f649146-86f7-45f2-b873-afc104a64c1c" (UID: "7f649146-86f7-45f2-b873-afc104a64c1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.147773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.147746    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-tmp-dir" pod "7f649146-86f7-45f2-b873-afc104a64c1c" (UID: "7f649146-86f7-45f2-b873-afc104a64c1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.148173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.148149    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.148263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.148176    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.148774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.148748    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-model-cache" pod "7f649146-86f7-45f2-b873-afc104a64c1c" (UID: "7f649146-86f7-45f2-b873-afc104a64c1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.150027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.150001    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-dshm" pod "7f649146-86f7-45f2-b873-afc104a64c1c" (UID: "7f649146-86f7-45f2-b873-afc104a64c1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.150634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.150508    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f649146-86f7-45f2-b873-afc104a64c1c-tls-certs" pod "7f649146-86f7-45f2-b873-afc104a64c1c" (UID: "7f649146-86f7-45f2-b873-afc104a64c1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:00.150634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.150596    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f649146-86f7-45f2-b873-afc104a64c1c-kube-api-access-t4gf9" pod "7f649146-86f7-45f2-b873-afc104a64c1c" (UID: "7f649146-86f7-45f2-b873-afc104a64c1c"). InnerVolumeSpecName "kube-api-access-t4gf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:00.188180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.187763    7019 generic.go:356] "Generic (PLEG): container finished" podID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerID="c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63" exitCode=0
Nov 02 15:49:00.188180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.187856    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" event={"ID":"61f22b8b-7228-4294-9c9b-f74ad0c5e98e","Type":"ContainerDied","Data":"c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63"}
Nov 02 15:49:00.188180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.187892    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr" event={"ID":"61f22b8b-7228-4294-9c9b-f74ad0c5e98e","Type":"ContainerDied","Data":"c9b0158df23a652a65fcc232e45a7abb2c9aa52f71a4abddf88ad03d9f2fa391"}
Nov 02 15:49:00.188180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.187912    7019 scope.go:122] "RemoveContainer" containerID="c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63"
Nov 02 15:49:00.188180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.188061    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"
Nov 02 15:49:00.194299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.193946    7019 generic.go:356] "Generic (PLEG): container finished" podID="7f649146-86f7-45f2-b873-afc104a64c1c" containerID="65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae" exitCode=0
Nov 02 15:49:00.194299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.194082    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" event={"ID":"7f649146-86f7-45f2-b873-afc104a64c1c","Type":"ContainerDied","Data":"65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae"}
Nov 02 15:49:00.194299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.194116    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58" event={"ID":"7f649146-86f7-45f2-b873-afc104a64c1c","Type":"ContainerDied","Data":"11104c2236eee683187fb68a176af9099a51f8fdcbec114653c1d4e8038e30dc"}
Nov 02 15:49:00.194299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.194200    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"
Nov 02 15:49:00.205289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.205121    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9" event={"ID":"b5375aab-a4a3-4285-b6a6-e0eeb3123a3b","Type":"ContainerDied","Data":"71c69e6697110d356bab0d58f2ae7586371e4a7d6abafa2127c34b5ef30a0d9e"}
Nov 02 15:49:00.205289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.205190    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71c69e6697110d356bab0d58f2ae7586371e4a7d6abafa2127c34b5ef30a0d9e"
Nov 02 15:49:00.205289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.205247    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"
Nov 02 15:49:00.217945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.217914    7019 scope.go:122] "RemoveContainer" containerID="c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63"
Nov 02 15:49:00.218744 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:49:00.218392    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63\": container with ID starting with c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63 not found: ID does not exist" containerID="c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63"
Nov 02 15:49:00.218744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.218429    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63"} err="failed to get container status \"c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63\": rpc error: code = NotFound desc = could not find container \"c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63\": container with ID starting with c0e1001a109abe63dbb10a859e52fbcc1d5e0149235d9480669d2ef54b1c3d63 not found: ID does not exist"
Nov 02 15:49:00.218744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.218456    7019 scope.go:122] "RemoveContainer" containerID="65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae"
Nov 02 15:49:00.221165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.221140    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr" event={"ID":"be94e6d6-dfc6-4cdb-9178-55cab00910e6","Type":"ContainerDied","Data":"58a9afd9339af0385d943be66027052404a548b5823c95d98bb32b71e322aea1"}
Nov 02 15:49:00.221493 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.221474    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a9afd9339af0385d943be66027052404a548b5823c95d98bb32b71e322aea1"
Nov 02 15:49:00.221604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.221358    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"
Nov 02 15:49:00.235463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.235441    7019 scope.go:122] "RemoveContainer" containerID="65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae"
Nov 02 15:49:00.235892 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:49:00.235825    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae\": container with ID starting with 65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae not found: ID does not exist" containerID="65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae"
Nov 02 15:49:00.235892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.235858    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae"} err="failed to get container status \"65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae\": rpc error: code = NotFound desc = could not find container \"65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae\": container with ID starting with 65d717cf05676fa8b0012afce25acaec94935a07a55091abb35f7687c91068ae not found: ID does not exist"
Nov 02 15:49:00.250484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.250457    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tls-certs\" (UniqueName: \"kubernetes.io/secret/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tls-certs\") pod \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") "
Nov 02 15:49:00.250735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.250713    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tmp-dir\") pod \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") "
Nov 02 15:49:00.250869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.250739    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-model-cache\") pod \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") "
Nov 02 15:49:00.250969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.250952    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tmp-dir" pod "61f22b8b-7228-4294-9c9b-f74ad0c5e98e" (UID: "61f22b8b-7228-4294-9c9b-f74ad0c5e98e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.251169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.251155    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-model-cache" pod "61f22b8b-7228-4294-9c9b-f74ad0c5e98e" (UID: "61f22b8b-7228-4294-9c9b-f74ad0c5e98e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.251235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.251208    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-dshm\") pod \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") "
Nov 02 15:49:00.251290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.251236    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-home\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-home\") pod \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") "
Nov 02 15:49:00.251569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.251555    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-kube-api-access-564bj\" (UniqueName: \"kubernetes.io/projected/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-kube-api-access-564bj\") pod \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\" (UID: \"61f22b8b-7228-4294-9c9b-f74ad0c5e98e\") "
Nov 02 15:49:00.251835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.251686    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-home" pod "61f22b8b-7228-4294-9c9b-f74ad0c5e98e" (UID: "61f22b8b-7228-4294-9c9b-f74ad0c5e98e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.252443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.252417    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f649146-86f7-45f2-b873-afc104a64c1c-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.252582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.252567    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4gf9\" (UniqueName: \"kubernetes.io/projected/7f649146-86f7-45f2-b873-afc104a64c1c-kube-api-access-t4gf9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.252704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.252689    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.252814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.252800    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.252905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.252895    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f649146-86f7-45f2-b873-afc104a64c1c-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.252992 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.252983    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.253079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.253069    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.253794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.253629    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-dshm" pod "61f22b8b-7228-4294-9c9b-f74ad0c5e98e" (UID: "61f22b8b-7228-4294-9c9b-f74ad0c5e98e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:00.266344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.266269    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"]
Nov 02 15:49:00.272250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.272215    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-w6-5113c2-kserve-84b6976588-29k58"]
Nov 02 15:49:00.278492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.278460    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tls-certs" pod "61f22b8b-7228-4294-9c9b-f74ad0c5e98e" (UID: "61f22b8b-7228-4294-9c9b-f74ad0c5e98e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:00.278666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.278610    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-kube-api-access-564bj" pod "61f22b8b-7228-4294-9c9b-f74ad0c5e98e" (UID: "61f22b8b-7228-4294-9c9b-f74ad0c5e98e"). InnerVolumeSpecName "kube-api-access-564bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:00.363497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.363286    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.363497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.363348    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.363497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.363365    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564bj\" (UniqueName: \"kubernetes.io/projected/61f22b8b-7228-4294-9c9b-f74ad0c5e98e-kube-api-access-564bj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:00.543576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.543543    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"]
Nov 02 15:49:00.557834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:00.557802    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5113c2/distinct-2-w6-5113c2-kserve-55b85877dc-dnqrr"]
Nov 02 15:49:01.733890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:01.733852    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:01.863169 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:01.863136    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" path="/var/lib/kubelet/pods/61f22b8b-7228-4294-9c9b-f74ad0c5e98e/volumes"
Nov 02 15:49:01.863671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:01.863642    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" path="/var/lib/kubelet/pods/7f649146-86f7-45f2-b873-afc104a64c1c/volumes"
Nov 02 15:49:03.797724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:03.797677    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:05.389775 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:05.389739    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:06.970490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:06.970450    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:08.609025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:08.608985    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:08.981151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:08.981119    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:10.005686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.005616    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"]
Nov 02 15:49:10.006964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.006930    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main"
Nov 02 15:49:10.006964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.006965    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main"
Nov 02 15:49:10.007184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.006997    7019 cpu_manager.go:414] "containerMap: removing container" podUID="be94e6d6-dfc6-4cdb-9178-55cab00910e6" containerName="revoke-keys"
Nov 02 15:49:10.007184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007006    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be94e6d6-dfc6-4cdb-9178-55cab00910e6" containerName="revoke-keys"
Nov 02 15:49:10.007184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007023    7019 cpu_manager.go:414] "containerMap: removing container" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main"
Nov 02 15:49:10.007184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007031    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main"
Nov 02 15:49:10.007184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007067    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" containerName="revoke-keys"
Nov 02 15:49:10.007184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007076    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" containerName="revoke-keys"
Nov 02 15:49:10.007529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007216    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f649146-86f7-45f2-b873-afc104a64c1c" containerName="main"
Nov 02 15:49:10.007529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007234    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="be94e6d6-dfc6-4cdb-9178-55cab00910e6" containerName="revoke-keys"
Nov 02 15:49:10.007529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007247    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f22b8b-7228-4294-9c9b-f74ad0c5e98e" containerName="main"
Nov 02 15:49:10.007529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.007261    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" containerName="revoke-keys"
Nov 02 15:49:10.015260 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.015234    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.042063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.042023    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"]
Nov 02 15:49:10.043634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.043607    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e40940-openshift-default-dockercfg-48t4g\""
Nov 02 15:49:10.070624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.070503    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:10.117026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.116990    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117311    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117356    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117501    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117562    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117704    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117761    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnsqp\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-kube-api-access-fnsqp\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117852    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.117938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.117913    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219128    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219204 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219201    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219257    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219298    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnsqp\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-kube-api-access-fnsqp\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219337    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219384    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219473 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219461    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219549    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.219778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.219598    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.220102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.220067    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.220496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.220474    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-certs\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.221039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.220776    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istiod-ca-cert\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.221220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.220906    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-credential-socket\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.221376 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.221206    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-data\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.223230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.223202    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-podinfo\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.231250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.231226    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-envoy\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.247988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.247917    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-token\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.249031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.249008    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnsqp\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-kube-api-access-fnsqp\") pod \"e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") " pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.293025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.292581    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:10.345115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.345081    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:10.447608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.440877    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:10.583725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.583333    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:10.772671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.755988    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:10.871777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.859786    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:10.871777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.859884    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:10.871777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.859926    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:10.876884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.876639    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"]
Nov 02 15:49:10.916926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:10.916894    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:11.044895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.044856    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"]
Nov 02 15:49:11.050299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.050271    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dd139-ngdqs"]
Nov 02 15:49:11.062764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.062720    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:11.320320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.320226    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" event={"ID":"e394f8f8-2b26-4dcd-8fa9-84a62560e269","Type":"ContainerStarted","Data":"2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81"}
Nov 02 15:49:11.320320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.320313    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" event={"ID":"e394f8f8-2b26-4dcd-8fa9-84a62560e269","Type":"ContainerStarted","Data":"29a1a132d7b2d7f32042d3b08279258df2dc1f41d753d343d9032689631d98e9"}
Nov 02 15:49:11.345485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.345453    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:11.347185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.347158    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Nov 02 15:49:11.347311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.347221    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Nov 02 15:49:11.376405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.376344    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podStartSLOduration=2.376328846 podStartE2EDuration="2.376328846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:11.372075818 +0000 UTC m=+3145.626029442" watchObservedRunningTime="2026-11-02 15:49:11.376328846 +0000 UTC m=+3145.630282470"
Nov 02 15:49:11.554549 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.554516    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:11.862582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:11.862543    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826d955c-0781-4315-baeb-31660f76591f" path="/var/lib/kubelet/pods/826d955c-0781-4315-baeb-31660f76591f/volumes"
Nov 02 15:49:12.031701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:12.031630    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"]
Nov 02 15:49:12.034845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:12.034821    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa75d81-9j7pg"]
Nov 02 15:49:12.345870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:12.345787    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Nov 02 15:49:12.345870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:12.345849    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Nov 02 15:49:12.746129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:12.746095    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:13.346348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:13.346315    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Nov 02 15:49:13.346763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:13.346363    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Nov 02 15:49:13.863052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:13.863016    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a0676e-8b8e-4b23-b0b8-6da21c75df2d" path="/var/lib/kubelet/pods/67a0676e-8b8e-4b23-b0b8-6da21c75df2d/volumes"
Nov 02 15:49:14.091681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:14.091609    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:14.345405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:14.345318    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Nov 02 15:49:14.345405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:14.345375    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Nov 02 15:49:14.510673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:14.504940    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:15.346173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:15.346134    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Nov 02 15:49:15.346386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:15.346195    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Nov 02 15:49:16.195360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:16.195314    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:16.346755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:16.346716    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Nov 02 15:49:16.346940 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:16.346771    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Nov 02 15:49:17.345988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:17.345939    7019 patch_prober.go:29] interesting pod/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body=
Nov 02 15:49:17.346513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:17.346012    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused"
Nov 02 15:49:17.838407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:17.838372    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:17.978337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:17.978293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:18.126057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.126020    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"]
Nov 02 15:49:18.126517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.126486    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy" containerID="cri-o://272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1" gracePeriod=30
Nov 02 15:49:18.187469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.187423    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:18.328809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.328772    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:18.351875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.351840    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:18.352278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.352126    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:18.352847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.352828    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:49:18.717633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.717598    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"]
Nov 02 15:49:18.721776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.721758    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.741282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.741248    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"]
Nov 02 15:49:18.791801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.791760    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:49:18.791983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.791808    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:49:18.813828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.813787    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.814018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.813860    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.814079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.814042    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2h5\" (UniqueName: \"kubernetes.io/projected/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-kube-api-access-jg2h5\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.915141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.915100    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2h5\" (UniqueName: \"kubernetes.io/projected/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-kube-api-access-jg2h5\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.915367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.915230    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.915367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.915266    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.916008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.915983    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.917689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.917662    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:18.941194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:18.941157    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2h5\" (UniqueName: \"kubernetes.io/projected/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-kube-api-access-jg2h5\") pod \"maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:19.037336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:19.037249    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:19.344281 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:49:19.344246    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af72b77_53a8_4b35_9b68_d9b4e1965ec1.slice/crio-b32a964ce2fca6b0e2160c6db2379ed4f4f02caee522d811b07d6a88401d721b WatchSource:0}: Error finding container b32a964ce2fca6b0e2160c6db2379ed4f4f02caee522d811b07d6a88401d721b: Status 404 returned error can't find the container with id b32a964ce2fca6b0e2160c6db2379ed4f4f02caee522d811b07d6a88401d721b
Nov 02 15:49:19.345202 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:19.345172    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"]
Nov 02 15:49:19.385440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:19.385413    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx" event={"ID":"3af72b77-53a8-4b35-9b68-d9b4e1965ec1","Type":"ContainerStarted","Data":"b32a964ce2fca6b0e2160c6db2379ed4f4f02caee522d811b07d6a88401d721b"}
Nov 02 15:49:20.062724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:20.059797    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:20.222794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:20.222729    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:20.395021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:20.394975    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx" event={"ID":"3af72b77-53a8-4b35-9b68-d9b4e1965ec1","Type":"ContainerStarted","Data":"2c94406c7fd744bf9286ffe37be08fcd3f4b0d9d232e46699def1d2d98b2ce91"}
Nov 02 15:49:20.450409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:20.450360    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx" podStartSLOduration=2.450347151 podStartE2EDuration="2.450347151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:49:20.448944244 +0000 UTC m=+3154.702897851" watchObservedRunningTime="2026-11-02 15:49:20.450347151 +0000 UTC m=+3154.704300775"
Nov 02 15:49:21.408077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:21.408032    7019 generic.go:356] "Generic (PLEG): container finished" podID="3af72b77-53a8-4b35-9b68-d9b4e1965ec1" containerID="2c94406c7fd744bf9286ffe37be08fcd3f4b0d9d232e46699def1d2d98b2ce91" exitCode=0
Nov 02 15:49:21.408491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:21.408113    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx" event={"ID":"3af72b77-53a8-4b35-9b68-d9b4e1965ec1","Type":"ContainerDied","Data":"2c94406c7fd744bf9286ffe37be08fcd3f4b0d9d232e46699def1d2d98b2ce91"}
Nov 02 15:49:21.442031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:21.441990    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:21.806856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:21.805460    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:21.981854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:21.981821    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:22.134030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.133994    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:22.737399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.737213    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:22.778693 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:49:22.776544    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:49:22.934888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.934862    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:22.989145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.989061    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-openshift-ca\") pod \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") "
Nov 02 15:49:22.989410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.989208    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-xks-ca\") pod \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") "
Nov 02 15:49:22.989410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.989290    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-kube-api-access-jg2h5\" (UniqueName: \"kubernetes.io/projected/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-kube-api-access-jg2h5\") pod \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\" (UID: \"3af72b77-53a8-4b35-9b68-d9b4e1965ec1\") "
Nov 02 15:49:22.989546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.989487    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-openshift-ca" pod "3af72b77-53a8-4b35-9b68-d9b4e1965ec1" (UID: "3af72b77-53a8-4b35-9b68-d9b4e1965ec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:22.989893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.989866    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:22.991850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.991822    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-xks-ca" pod "3af72b77-53a8-4b35-9b68-d9b4e1965ec1" (UID: "3af72b77-53a8-4b35-9b68-d9b4e1965ec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:22.992145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:22.992119    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-kube-api-access-jg2h5" pod "3af72b77-53a8-4b35-9b68-d9b4e1965ec1" (UID: "3af72b77-53a8-4b35-9b68-d9b4e1965ec1"). InnerVolumeSpecName "kube-api-access-jg2h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:23.035700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.035663    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"]
Nov 02 15:49:23.039540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.039513    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01daa0fe-8hhfz"]
Nov 02 15:49:23.090984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.090948    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2h5\" (UniqueName: \"kubernetes.io/projected/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-kube-api-access-jg2h5\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.090984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.090983    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3af72b77-53a8-4b35-9b68-d9b4e1965ec1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.386688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.386631    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:49:23.395282 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395247    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-podinfo\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395297    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-certs\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395357    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-envoy\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395478    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-token\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-token\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395621 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395553    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-socket\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395673    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-certs" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:23.395767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395706    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-credential-socket\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395768    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istiod-ca-cert\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395801    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-kube-api-access-q4c5p\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-kube-api-access-q4c5p\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.395916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395820    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-socket" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:23.395916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395841    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-data\") pod \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\" (UID: \"2eedea98-57e6-4ca6-bdf0-39e3dc001705\") "
Nov 02 15:49:23.396187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.395979    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-credential-socket" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:23.396187 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.396040    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-data" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:23.396315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.396221    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istiod-ca-cert" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:23.396617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.396596    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.396686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.396626    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.396686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.396640    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.396762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.396692    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.396762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.396705    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.397722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.397699    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-podinfo" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:49:23.397955 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.397937    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-envoy" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:23.398146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.398127    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-token" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:23.398217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.398172    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-kube-api-access-q4c5p" pod "2eedea98-57e6-4ca6-bdf0-39e3dc001705" (UID: "2eedea98-57e6-4ca6-bdf0-39e3dc001705"). InnerVolumeSpecName "kube-api-access-q4c5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:23.466898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.466856    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx" event={"ID":"3af72b77-53a8-4b35-9b68-d9b4e1965ec1","Type":"ContainerDied","Data":"b32a964ce2fca6b0e2160c6db2379ed4f4f02caee522d811b07d6a88401d721b"}
Nov 02 15:49:23.466898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.466883    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"
Nov 02 15:49:23.467148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.466926    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b32a964ce2fca6b0e2160c6db2379ed4f4f02caee522d811b07d6a88401d721b"
Nov 02 15:49:23.469080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.469055    7019 generic.go:356] "Generic (PLEG): container finished" podID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerID="272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1" exitCode=0
Nov 02 15:49:23.469200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.469097    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"
Nov 02 15:49:23.469200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.469137    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" event={"ID":"2eedea98-57e6-4ca6-bdf0-39e3dc001705","Type":"ContainerDied","Data":"272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1"}
Nov 02 15:49:23.469200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.469189    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf" event={"ID":"2eedea98-57e6-4ca6-bdf0-39e3dc001705","Type":"ContainerDied","Data":"609fb0292ff0c9672ed727c003d01b36e635f6940d62e39f9d9cfe9196495343"}
Nov 02 15:49:23.469377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.469219    7019 scope.go:122] "RemoveContainer" containerID="272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1"
Nov 02 15:49:23.484404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.484365    7019 scope.go:122] "RemoveContainer" containerID="26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0"
Nov 02 15:49:23.497842 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.497809    7019 scope.go:122] "RemoveContainer" containerID="272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1"
Nov 02 15:49:23.498230 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:49:23.498209    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1\": container with ID starting with 272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1 not found: ID does not exist" containerID="272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1"
Nov 02 15:49:23.498343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.498240    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1"} err="failed to get container status \"272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1\": rpc error: code = NotFound desc = could not find container \"272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1\": container with ID starting with 272d2c92ad1f75272719c00c79e2a1cfd8f54cc6f8816c2451426a933351cae1 not found: ID does not exist"
Nov 02 15:49:23.498343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.498259    7019 scope.go:122] "RemoveContainer" containerID="26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0"
Nov 02 15:49:23.498549 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:49:23.498531    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0\": container with ID starting with 26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0 not found: ID does not exist" containerID="26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0"
Nov 02 15:49:23.498608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.498554    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0"} err="failed to get container status \"26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0\": rpc error: code = NotFound desc = could not find container \"26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0\": container with ID starting with 26df06316fd7a001ebfa8e09dbe1aea39c952ddbdc27dca5a627c8cb142f09a0 not found: ID does not exist"
Nov 02 15:49:23.498771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.498751    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.498848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.498777    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.498848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.498790    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4c5p\" (UniqueName: \"kubernetes.io/projected/2eedea98-57e6-4ca6-bdf0-39e3dc001705-kube-api-access-q4c5p\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.498848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.498804    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eedea98-57e6-4ca6-bdf0-39e3dc001705-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:23.525731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.525703    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:23.532096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.532073    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"]
Nov 02 15:49:23.537303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.537274    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5dd139-openshift-default-5c59c49cf-2crkf"]
Nov 02 15:49:23.671257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.671221    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:23.863165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.863126    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" path="/var/lib/kubelet/pods/2eedea98-57e6-4ca6-bdf0-39e3dc001705/volumes"
Nov 02 15:49:23.863831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:23.863811    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d6f4a7-4420-496e-96a1-996618c94d5e" path="/var/lib/kubelet/pods/98d6f4a7-4420-496e-96a1-996618c94d5e/volumes"
Nov 02 15:49:25.295581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:25.295547    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:28.757217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:28.757134    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:28.906586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:28.906553    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:29.591774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:29.591738    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:29.804409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:29.804377    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:29.973263 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:29.973225    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:30.114243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:30.114209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:31.567233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:31.567150    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:31.736326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:31.736293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:31.893133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:31.893101    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:32.110097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:32.110031    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:32.210633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:32.210544    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"]
Nov 02 15:49:32.221993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:32.221958    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-24255b-l58bx"]
Nov 02 15:49:32.274151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:32.274106    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:32.771570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:32.771527    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:33.862622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:33.862587    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af72b77-53a8-4b35-9b68-d9b4e1965ec1" path="/var/lib/kubelet/pods/3af72b77-53a8-4b35-9b68-d9b4e1965ec1/volumes"
Nov 02 15:49:34.006138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:34.006094    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:35.429813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:35.429774    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"]
Nov 02 15:49:35.430440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:35.430036    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs" podUID="65d9ead9-55e5-46cb-85d9-e837fcdf8716" containerName="maas-api" containerID="cri-o://a81adde0c44e7dfcafc6fcd45c5898d1d7dff8ba55d408c89467996c13b4900e" gracePeriod=30
Nov 02 15:49:35.544462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:35.544425    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:49:35.546987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:35.546955    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:35.568853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:35.568816    7019 generic.go:356] "Generic (PLEG): container finished" podID="65d9ead9-55e5-46cb-85d9-e837fcdf8716" containerID="a81adde0c44e7dfcafc6fcd45c5898d1d7dff8ba55d408c89467996c13b4900e" exitCode=0
Nov 02 15:49:35.569028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:35.569001    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs" event={"ID":"65d9ead9-55e5-46cb-85d9-e837fcdf8716","Type":"ContainerDied","Data":"a81adde0c44e7dfcafc6fcd45c5898d1d7dff8ba55d408c89467996c13b4900e"}
Nov 02 15:49:36.038964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.038932    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:49:36.095390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.095315    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:36.097751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.097710    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d9ead9-55e5-46cb-85d9-e837fcdf8716-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d9ead9-55e5-46cb-85d9-e837fcdf8716-openshift-service-ca\") pod \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") "
Nov 02 15:49:36.098006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.097978    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d9ead9-55e5-46cb-85d9-e837fcdf8716-kube-api-access-hjmh4\" (UniqueName: \"kubernetes.io/projected/65d9ead9-55e5-46cb-85d9-e837fcdf8716-kube-api-access-hjmh4\") pod \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") "
Nov 02 15:49:36.098408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.098382    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-metrics-certs\") pod \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") "
Nov 02 15:49:36.098551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.098537    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-maas-api-tls\") pod \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\" (UID: \"65d9ead9-55e5-46cb-85d9-e837fcdf8716\") "
Nov 02 15:49:36.104144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.104086    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d9ead9-55e5-46cb-85d9-e837fcdf8716-openshift-service-ca" pod "65d9ead9-55e5-46cb-85d9-e837fcdf8716" (UID: "65d9ead9-55e5-46cb-85d9-e837fcdf8716"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:36.126096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.126035    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d9ead9-55e5-46cb-85d9-e837fcdf8716-kube-api-access-hjmh4" pod "65d9ead9-55e5-46cb-85d9-e837fcdf8716" (UID: "65d9ead9-55e5-46cb-85d9-e837fcdf8716"). InnerVolumeSpecName "kube-api-access-hjmh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:36.142384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.142299    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-maas-api-tls" pod "65d9ead9-55e5-46cb-85d9-e837fcdf8716" (UID: "65d9ead9-55e5-46cb-85d9-e837fcdf8716"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:36.158875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.158776    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-metrics-certs" pod "65d9ead9-55e5-46cb-85d9-e837fcdf8716" (UID: "65d9ead9-55e5-46cb-85d9-e837fcdf8716"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:36.201675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.198693    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"]
Nov 02 15:49:36.205709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.204813    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65d9ead9-55e5-46cb-85d9-e837fcdf8716-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:36.205709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.204846    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjmh4\" (UniqueName: \"kubernetes.io/projected/65d9ead9-55e5-46cb-85d9-e837fcdf8716-kube-api-access-hjmh4\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:36.205709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.204866    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:36.205709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.204879    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65d9ead9-55e5-46cb-85d9-e837fcdf8716-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:36.205709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.205457    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-5113c2-srpzs"]
Nov 02 15:49:36.238965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.238929    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:36.385541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.385497    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:36.590222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.590181    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs" event={"ID":"65d9ead9-55e5-46cb-85d9-e837fcdf8716","Type":"ContainerDied","Data":"9d38c7ea4bf304f9ddcbe5dfcb168ff974aa9df49a9d9ebb1ff69e08ce1ceb9c"}
Nov 02 15:49:36.590222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.590215    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"
Nov 02 15:49:36.590826 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.590270    7019 scope.go:122] "RemoveContainer" containerID="a81adde0c44e7dfcafc6fcd45c5898d1d7dff8ba55d408c89467996c13b4900e"
Nov 02 15:49:36.685115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.685080    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:36.710741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.710703    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"]
Nov 02 15:49:36.731682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:36.727898    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-5113c2-5d8545899b-jmbzs"]
Nov 02 15:49:37.863804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:37.863757    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b465de7-2a1a-41f3-9c49-18a12fe2c596" path="/var/lib/kubelet/pods/3b465de7-2a1a-41f3-9c49-18a12fe2c596/volumes"
Nov 02 15:49:37.864438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:37.864415    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d9ead9-55e5-46cb-85d9-e837fcdf8716" path="/var/lib/kubelet/pods/65d9ead9-55e5-46cb-85d9-e837fcdf8716/volumes"
Nov 02 15:49:38.433918 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:38.433881    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:38.595168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:38.595116    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:38.755755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:38.755664    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:39.857439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:39.857403    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:40.011861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.011823    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:40.162675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.152639    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:40.314674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.312212    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:40.729295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729257    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"]
Nov 02 15:49:40.729895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729880    7019 cpu_manager.go:414] "containerMap: removing container" podUID="65d9ead9-55e5-46cb-85d9-e837fcdf8716" containerName="maas-api"
Nov 02 15:49:40.729946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729896    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d9ead9-55e5-46cb-85d9-e837fcdf8716" containerName="maas-api"
Nov 02 15:49:40.729946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729906    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy"
Nov 02 15:49:40.729946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729911    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy"
Nov 02 15:49:40.729946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729919    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy"
Nov 02 15:49:40.729946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729925    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy"
Nov 02 15:49:40.729946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729934    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3af72b77-53a8-4b35-9b68-d9b4e1965ec1" containerName="revoke-keys"
Nov 02 15:49:40.729946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.729939    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af72b77-53a8-4b35-9b68-d9b4e1965ec1" containerName="revoke-keys"
Nov 02 15:49:40.730182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.730052    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d9ead9-55e5-46cb-85d9-e837fcdf8716" containerName="maas-api"
Nov 02 15:49:40.730182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.730060    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy"
Nov 02 15:49:40.730182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.730067    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eedea98-57e6-4ca6-bdf0-39e3dc001705" containerName="istio-proxy"
Nov 02 15:49:40.730182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.730074    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af72b77-53a8-4b35-9b68-d9b4e1965ec1" containerName="revoke-keys"
Nov 02 15:49:40.734858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.734837    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.761606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.761550    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"]
Nov 02 15:49:40.834397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.833619    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.834397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.833711    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.834397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.833901    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc78n\" (UniqueName: \"kubernetes.io/projected/0822e3ef-efde-4bd8-8b81-2816fc80e607-kube-api-access-wc78n\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.839897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.839742    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:49:40.839897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.839848    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:49:40.935037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.934996    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc78n\" (UniqueName: \"kubernetes.io/projected/0822e3ef-efde-4bd8-8b81-2816fc80e607-kube-api-access-wc78n\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.935533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.935165    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.935533 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.935212    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.935913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.935890    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.937879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.937853    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:40.960597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:40.960560    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc78n\" (UniqueName: \"kubernetes.io/projected/0822e3ef-efde-4bd8-8b81-2816fc80e607-kube-api-access-wc78n\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:41.053184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:41.053105    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:41.406294 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:49:41.406250    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0822e3ef_efde_4bd8_8b81_2816fc80e607.slice/crio-c7955644e51873ae906954f0adcce6e72ce9a08eef5e20874b294c48426454e0 WatchSource:0}: Error finding container c7955644e51873ae906954f0adcce6e72ce9a08eef5e20874b294c48426454e0: Status 404 returned error can't find the container with id c7955644e51873ae906954f0adcce6e72ce9a08eef5e20874b294c48426454e0
Nov 02 15:49:41.406571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:41.406551    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"]
Nov 02 15:49:41.636165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:41.636131    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9" event={"ID":"0822e3ef-efde-4bd8-8b81-2816fc80e607","Type":"ContainerStarted","Data":"c7955644e51873ae906954f0adcce6e72ce9a08eef5e20874b294c48426454e0"}
Nov 02 15:49:41.832993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:41.832952    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:42.645230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:42.645193    7019 generic.go:356] "Generic (PLEG): container finished" podID="0822e3ef-efde-4bd8-8b81-2816fc80e607" containerID="2adcd2af02395ed6cebe8475aaa00e4db1f3a09e658dcec5083a3e665ee9571b" exitCode=0
Nov 02 15:49:42.645643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:42.645272    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9" event={"ID":"0822e3ef-efde-4bd8-8b81-2816fc80e607","Type":"ContainerDied","Data":"2adcd2af02395ed6cebe8475aaa00e4db1f3a09e658dcec5083a3e665ee9571b"}
Nov 02 15:49:42.908932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:42.908889    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:43.062841 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:43.062793    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:43.148673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:43.145339    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"]
Nov 02 15:49:43.148673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:43.145591    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" containerID="cri-o://b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0" gracePeriod=30
Nov 02 15:49:43.213042 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:43.212941    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:43.794119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:43.794058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:43.895902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:43.895873    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:44.010338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.010240    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0822e3ef-efde-4bd8-8b81-2816fc80e607-kube-api-access-wc78n\" (UniqueName: \"kubernetes.io/projected/0822e3ef-efde-4bd8-8b81-2816fc80e607-kube-api-access-wc78n\") pod \"0822e3ef-efde-4bd8-8b81-2816fc80e607\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") "
Nov 02 15:49:44.010520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.010384    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-openshift-ca\") pod \"0822e3ef-efde-4bd8-8b81-2816fc80e607\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") "
Nov 02 15:49:44.010520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.010452    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-xks-ca\") pod \"0822e3ef-efde-4bd8-8b81-2816fc80e607\" (UID: \"0822e3ef-efde-4bd8-8b81-2816fc80e607\") "
Nov 02 15:49:44.011039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.011008    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-openshift-ca" pod "0822e3ef-efde-4bd8-8b81-2816fc80e607" (UID: "0822e3ef-efde-4bd8-8b81-2816fc80e607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:44.011400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.011372    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:44.012907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.012880    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0822e3ef-efde-4bd8-8b81-2816fc80e607-kube-api-access-wc78n" pod "0822e3ef-efde-4bd8-8b81-2816fc80e607" (UID: "0822e3ef-efde-4bd8-8b81-2816fc80e607"). InnerVolumeSpecName "kube-api-access-wc78n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:44.013311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.013289    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-xks-ca" pod "0822e3ef-efde-4bd8-8b81-2816fc80e607" (UID: "0822e3ef-efde-4bd8-8b81-2816fc80e607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:49:44.113072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.113000    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc78n\" (UniqueName: \"kubernetes.io/projected/0822e3ef-efde-4bd8-8b81-2816fc80e607-kube-api-access-wc78n\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:44.113072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.113039    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0822e3ef-efde-4bd8-8b81-2816fc80e607-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:44.662835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.662803    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9" event={"ID":"0822e3ef-efde-4bd8-8b81-2816fc80e607","Type":"ContainerDied","Data":"c7955644e51873ae906954f0adcce6e72ce9a08eef5e20874b294c48426454e0"}
Nov 02 15:49:44.662835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.662821    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"
Nov 02 15:49:44.663076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:44.662853    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7955644e51873ae906954f0adcce6e72ce9a08eef5e20874b294c48426454e0"
Nov 02 15:49:45.904805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:45.904762    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:46.247443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:46.247360    7019 patch_prober.go:29] interesting pod/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:49:46.247443 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:46.247425    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:49:47.487352 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:47.487302    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:47.701531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:47.701491    7019 generic.go:356] "Generic (PLEG): container finished" podID="e77765a7-6626-4005-b10c-db1068f65c21" containerID="d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891" exitCode=0
Nov 02 15:49:47.701781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:47.701562    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" event={"ID":"e77765a7-6626-4005-b10c-db1068f65c21","Type":"ContainerDied","Data":"d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891"}
Nov 02 15:49:47.702052 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:47.702035    7019 scope.go:122] "RemoveContainer" containerID="d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891"
Nov 02 15:49:47.704008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:47.703974    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:47.704139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:47.704063    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:47.704139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:47.704114    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:49:48.418132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.418056    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476095    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-podinfo\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476201    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-token\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-token\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476307    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istiod-ca-cert\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476409    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-kube-api-access-hvfd9\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-kube-api-access-hvfd9\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476455    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-certs\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476515    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-socket\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476551    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-envoy\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476640    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-credential-socket\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.476787    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-data\") pod \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\" (UID: \"fa53c43b-3c4c-47d1-b98c-aca467ff95a0\") "
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.481749    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-data" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.485191    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-certs" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:48.486671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.486627    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-socket" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:48.487951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.486710    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-credential-socket" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:48.487951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.487081    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istiod-ca-cert" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:49:48.509564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.498234    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-envoy" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:49:48.511134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.510729    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-token" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:48.511134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.511095    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-kube-api-access-hvfd9" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "kube-api-access-hvfd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:49:48.511553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.511518    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-podinfo" pod "fa53c43b-3c4c-47d1-b98c-aca467ff95a0" (UID: "fa53c43b-3c4c-47d1-b98c-aca467ff95a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584789    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584832    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584849    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvfd9\" (UniqueName: \"kubernetes.io/projected/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-kube-api-access-hvfd9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584863    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584879    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584892    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584905    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584918    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.587671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.584933    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa53c43b-3c4c-47d1-b98c-aca467ff95a0-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:49:48.713115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.713015    7019 generic.go:356] "Generic (PLEG): container finished" podID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerID="b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0" exitCode=0
Nov 02 15:49:48.713115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.713067    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"
Nov 02 15:49:48.713115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.713095    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" event={"ID":"fa53c43b-3c4c-47d1-b98c-aca467ff95a0","Type":"ContainerDied","Data":"b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0"}
Nov 02 15:49:48.713424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.713155    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk" event={"ID":"fa53c43b-3c4c-47d1-b98c-aca467ff95a0","Type":"ContainerDied","Data":"10d04d575b2b30b8165ec08376ea0892277fa4bd3ca13e333e2fa9aae6cb4f5b"}
Nov 02 15:49:48.713424 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.713188    7019 scope.go:122] "RemoveContainer" containerID="b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0"
Nov 02 15:49:48.715373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.715354    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" event={"ID":"e77765a7-6626-4005-b10c-db1068f65c21","Type":"ContainerStarted","Data":"b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d"}
Nov 02 15:49:48.729214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.729178    7019 scope.go:122] "RemoveContainer" containerID="b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0"
Nov 02 15:49:48.729422 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:49:48.729403    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0\": container with ID starting with b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0 not found: ID does not exist" containerID="b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0"
Nov 02 15:49:48.729488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.729431    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0"} err="failed to get container status \"b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0\": rpc error: code = NotFound desc = could not find container \"b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0\": container with ID starting with b3b6d733269bf8d5948bf984b5352fd10ea7b5446ca9ab57ec11a3c3e730fea0 not found: ID does not exist"
Nov 02 15:49:48.842358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.842322    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"]
Nov 02 15:49:48.845960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:48.845935    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5113c2-openshift-default-8cb5b6fcf-fc5zk"]
Nov 02 15:49:49.123477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:49.123446    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:49:49.123972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:49.123953    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Nov 02 15:49:49.124023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:49.123996    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Nov 02 15:49:49.145116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:49.145087    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:49.862133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:49.862098    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" path="/var/lib/kubelet/pods/fa53c43b-3c4c-47d1-b98c-aca467ff95a0/volumes"
Nov 02 15:49:50.123577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:50.123482    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Nov 02 15:49:50.123577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:50.123558    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Nov 02 15:49:50.288673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:50.284209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:50.508719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:50.506133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:51.123406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:51.123367    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Nov 02 15:49:51.123838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:51.123424    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Nov 02 15:49:51.306090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:51.306048    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:51.455400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:51.455302    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:51.600689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:51.600568    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:52.133672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:52.132756    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Nov 02 15:49:52.133672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:52.132844    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Nov 02 15:49:52.706726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:52.706678    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:52.744914 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:49:52.743036    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:49:53.135100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.133740    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Nov 02 15:49:53.135100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.133817    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Nov 02 15:49:53.302199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.302002    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:53.385736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.366271    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"]
Nov 02 15:49:53.424527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.424492    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-246137-xdwd9"]
Nov 02 15:49:53.444960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.444931    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:53.861711 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.861684    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0822e3ef-efde-4bd8-8b81-2816fc80e607" path="/var/lib/kubelet/pods/0822e3ef-efde-4bd8-8b81-2816fc80e607/volumes"
Nov 02 15:49:53.891104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:53.891065    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:54.125545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:54.125452    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body=
Nov 02 15:49:54.125839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:54.125808    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused"
Nov 02 15:49:54.988290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:54.988256    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:49:55.019453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:55.019408    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:55.128971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:55.128941    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:49:55.560245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:55.560203    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:55.928504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:55.928473    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:49:56.796760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:56.796719    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:49:58.926834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:49:58.926796    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:00.027227 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:00.027190    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:00.210824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:00.206490    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:00.600910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:00.600875    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:00.739858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:00.739825    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:00.902639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:00.902593    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"]
Nov 02 15:50:00.902927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:00.902897    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy" containerID="cri-o://d6cd9fb72fb3721c9313f4ba52856c897e44f2b36a0b21dcbabd0c9139979613" gracePeriod=30
Nov 02 15:50:00.948050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:00.948017    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:01.098456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.098421    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:01.455452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.455418    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"]
Nov 02 15:50:01.456066 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.456052    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0822e3ef-efde-4bd8-8b81-2816fc80e607" containerName="revoke-keys"
Nov 02 15:50:01.456122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.456067    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0822e3ef-efde-4bd8-8b81-2816fc80e607" containerName="revoke-keys"
Nov 02 15:50:01.456122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.456082    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy"
Nov 02 15:50:01.456122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.456087    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy"
Nov 02 15:50:01.456234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.456183    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0822e3ef-efde-4bd8-8b81-2816fc80e607" containerName="revoke-keys"
Nov 02 15:50:01.456234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.456191    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa53c43b-3c4c-47d1-b98c-aca467ff95a0" containerName="istio-proxy"
Nov 02 15:50:01.461890 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.461871    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.479715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.479686    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"]
Nov 02 15:50:01.571968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.571933    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:50:01.572146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.572001    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:50:01.580639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.580614    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.580811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.580794    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlkw\" (UniqueName: \"kubernetes.io/projected/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-kube-api-access-2mlkw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.580948 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.580919    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.682852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.682811    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlkw\" (UniqueName: \"kubernetes.io/projected/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-kube-api-access-2mlkw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.683049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.682946    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.683122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.683063    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.683614 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.683594    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.685382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.685357    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.709682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.709607    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlkw\" (UniqueName: \"kubernetes.io/projected/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-kube-api-access-2mlkw\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:01.776415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:01.776384    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:02.084724 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:50:02.084689    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4a7715_c2aa_4c2f_bf32_44f6227b7d3f.slice/crio-0380926828ba056deccfbf5a4379b6cb4cf64da63df5200a7e780f0a5c533f0b WatchSource:0}: Error finding container 0380926828ba056deccfbf5a4379b6cb4cf64da63df5200a7e780f0a5c533f0b: Status 404 returned error can't find the container with id 0380926828ba056deccfbf5a4379b6cb4cf64da63df5200a7e780f0a5c533f0b
Nov 02 15:50:02.085388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:02.085362    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"]
Nov 02 15:50:02.879748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:02.879702    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:03.075827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:03.075787    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh" event={"ID":"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f","Type":"ContainerStarted","Data":"0dedef7b3f2ec9205f8cc64668e04f4d3c5fbe79fe9d8d17ed785c5514d14f2e"}
Nov 02 15:50:03.076015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:03.075852    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh" event={"ID":"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f","Type":"ContainerStarted","Data":"0380926828ba056deccfbf5a4379b6cb4cf64da63df5200a7e780f0a5c533f0b"}
Nov 02 15:50:03.121532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:03.121486    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh" podStartSLOduration=2.121474345 podStartE2EDuration="2.121474345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:03.120810898 +0000 UTC m=+3197.374764522" watchObservedRunningTime="2026-11-02 15:50:03.121474345 +0000 UTC m=+3197.375427969"
Nov 02 15:50:04.087763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:04.087719    7019 generic.go:356] "Generic (PLEG): container finished" podID="4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" containerID="0dedef7b3f2ec9205f8cc64668e04f4d3c5fbe79fe9d8d17ed785c5514d14f2e" exitCode=0
Nov 02 15:50:04.088196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:04.087795    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh" event={"ID":"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f","Type":"ContainerDied","Data":"0dedef7b3f2ec9205f8cc64668e04f4d3c5fbe79fe9d8d17ed785c5514d14f2e"}
Nov 02 15:50:04.577383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:04.577339    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:05.250815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.250790    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:05.357877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.357777    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-kube-api-access-2mlkw\" (UniqueName: \"kubernetes.io/projected/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-kube-api-access-2mlkw\") pod \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") "
Nov 02 15:50:05.358067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.357900    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-openshift-ca\") pod \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") "
Nov 02 15:50:05.358067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.358023    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-xks-ca\") pod \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\" (UID: \"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f\") "
Nov 02 15:50:05.358279 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.358255    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-openshift-ca" pod "4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" (UID: "4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:05.358595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.358576    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:05.360100 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.360074    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-xks-ca" pod "4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" (UID: "4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:50:05.360212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.360163    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-kube-api-access-2mlkw" pod "4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" (UID: "4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f"). InnerVolumeSpecName "kube-api-access-2mlkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:05.460205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.460167    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:05.460205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:05.460198    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlkw\" (UniqueName: \"kubernetes.io/projected/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f-kube-api-access-2mlkw\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.107542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.107453    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh" event={"ID":"4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f","Type":"ContainerDied","Data":"0380926828ba056deccfbf5a4379b6cb4cf64da63df5200a7e780f0a5c533f0b"}
Nov 02 15:50:06.107542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.107528    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0380926828ba056deccfbf5a4379b6cb4cf64da63df5200a7e780f0a5c533f0b"
Nov 02 15:50:06.107903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.107618    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"
Nov 02 15:50:06.111534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.111150    7019 generic.go:356] "Generic (PLEG): container finished" podID="e55f9fad-b571-44b9-97ea-145f877a4905" containerID="d6cd9fb72fb3721c9313f4ba52856c897e44f2b36a0b21dcbabd0c9139979613" exitCode=0
Nov 02 15:50:06.111534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.111424    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" event={"ID":"e55f9fad-b571-44b9-97ea-145f877a4905","Type":"ContainerDied","Data":"d6cd9fb72fb3721c9313f4ba52856c897e44f2b36a0b21dcbabd0c9139979613"}
Nov 02 15:50:06.151959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.151931    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:50:06.271394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271362    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-credential-socket\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271485    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-kube-api-access-hl2zj\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-kube-api-access-hl2zj\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271518    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-istio-token\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-istio-token\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271614    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e55f9fad-b571-44b9-97ea-145f877a4905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55f9fad-b571-44b9-97ea-145f877a4905-istio-podinfo\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271680    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55f9fad-b571-44b9-97ea-145f877a4905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55f9fad-b571-44b9-97ea-145f877a4905-istiod-ca-cert\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271700    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-credential-socket" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271733    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-envoy\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271854    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-certs\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.271911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271892    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-socket\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.272362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.271960    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-data\") pod \"e55f9fad-b571-44b9-97ea-145f877a4905\" (UID: \"e55f9fad-b571-44b9-97ea-145f877a4905\") "
Nov 02 15:50:06.272547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.272521    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55f9fad-b571-44b9-97ea-145f877a4905-istiod-ca-cert" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:06.272634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.272584    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-socket" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:50:06.272743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.272667    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.272743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.272640    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-certs" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:50:06.272843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.272795    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-data" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:50:06.275005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.274983    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e55f9fad-b571-44b9-97ea-145f877a4905-istio-podinfo" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:50:06.275127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.275009    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-envoy" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:50:06.275239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.275212    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-istio-token" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:06.275392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.275374    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-kube-api-access-hl2zj" pod "e55f9fad-b571-44b9-97ea-145f877a4905" (UID: "e55f9fad-b571-44b9-97ea-145f877a4905"). InnerVolumeSpecName "kube-api-access-hl2zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:06.317872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.317835    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:06.374385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374353    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl2zj\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-kube-api-access-hl2zj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.374385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374384    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e55f9fad-b571-44b9-97ea-145f877a4905-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.374385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374393    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e55f9fad-b571-44b9-97ea-145f877a4905-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.374658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374401    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e55f9fad-b571-44b9-97ea-145f877a4905-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.374658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374409    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.374658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374417    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.374658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374425    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:06.374658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:06.374433    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e55f9fad-b571-44b9-97ea-145f877a4905-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:07.120372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.120336    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"
Nov 02 15:50:07.120561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.120376    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm" event={"ID":"e55f9fad-b571-44b9-97ea-145f877a4905","Type":"ContainerDied","Data":"f4667c599aae77ffcff2aef9fa399faf30fd4694dcc32d6a5a670232dc8acf55"}
Nov 02 15:50:07.120561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.120452    7019 scope.go:122] "RemoveContainer" containerID="d6cd9fb72fb3721c9313f4ba52856c897e44f2b36a0b21dcbabd0c9139979613"
Nov 02 15:50:07.191576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.191537    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"]
Nov 02 15:50:07.202463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.202435    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-246137-openshift-default-6ff5dc8bbc-m22mm"]
Nov 02 15:50:07.608700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.608579    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:07.759399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.759357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:07.862120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.862092    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" path="/var/lib/kubelet/pods/e55f9fad-b571-44b9-97ea-145f877a4905/volumes"
Nov 02 15:50:07.985012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:07.984978    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:08.133766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.133638    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:08.405722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.405177    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"]
Nov 02 15:50:08.409522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.406153    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy"
Nov 02 15:50:08.409522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.406180    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy"
Nov 02 15:50:08.409522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.406200    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" containerName="revoke-keys"
Nov 02 15:50:08.409522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.406209    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" containerName="revoke-keys"
Nov 02 15:50:08.409522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.406325    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55f9fad-b571-44b9-97ea-145f877a4905" containerName="istio-proxy"
Nov 02 15:50:08.409522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.406342    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" containerName="revoke-keys"
Nov 02 15:50:08.412478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.412456    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.447961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.446819    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"]
Nov 02 15:50:08.455287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.455243    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a433eb-openshift-default-dockercfg-xrgfp\""
Nov 02 15:50:08.504269 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.503988    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:08.515332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515294    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-data\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515399    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-podinfo\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515460    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9klv\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-kube-api-access-v9klv\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515485    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d220ea-b8e7-492d-9036-68c13c08ece9-istiod-ca-cert\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515522    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-envoy\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515631    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-certs\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515674    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-credential-socket\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515694    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-token\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.515982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.515839    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-socket\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.617513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.617462    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-certs\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.626671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.618135    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-credential-socket\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.626877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.626799    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-token\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.626954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.626895    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-socket\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.627060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.627040    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-data\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.627139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.627122    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-podinfo\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.627231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.627213    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9klv\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-kube-api-access-v9klv\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.627301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.627260    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d220ea-b8e7-492d-9036-68c13c08ece9-istiod-ca-cert\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.627301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.627288    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-envoy\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.630734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.629964    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-socket\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.630734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.620418    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-certs\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.630734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.621252    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-credential-socket\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.632144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.632085    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d220ea-b8e7-492d-9036-68c13c08ece9-istiod-ca-cert\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.632540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.632520    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-data\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.658924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.648961    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-envoy\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.665325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.665286    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:08.684388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.684269    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-podinfo\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.703172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.703104    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-token\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.712671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.709328    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9klv\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-kube-api-access-v9klv\") pod \"e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") " pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:08.757672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:08.746736    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:09.111754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:09.110917    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:09.168232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:09.162557    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"]
Nov 02 15:50:09.168232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:09.165741    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:50:09.168232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:09.165826    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:50:09.168232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:09.165868    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:50:09.709671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:09.709593    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:10.145481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:10.145438    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" event={"ID":"89d220ea-b8e7-492d-9036-68c13c08ece9","Type":"ContainerStarted","Data":"e624d8538a0cfe9baaf4e3dac96c76b088720eac9e10c8ce91032d62d5befc34"}
Nov 02 15:50:10.145693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:10.145510    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" event={"ID":"89d220ea-b8e7-492d-9036-68c13c08ece9","Type":"ContainerStarted","Data":"4791a49ecc648533a791a85fa8f13de3497a25385f936e9d9c1a22e55a410302"}
Nov 02 15:50:10.205885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:10.205834    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" podStartSLOduration=2.205818429 podStartE2EDuration="2.205818429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:10.20556417 +0000 UTC m=+3204.459517795" watchObservedRunningTime="2026-11-02 15:50:10.205818429 +0000 UTC m=+3204.459772056"
Nov 02 15:50:10.761477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:10.761443    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:10.767904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:10.767875    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:11.152910 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:11.152878    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:11.153703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:11.153681    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:50:11.283953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:11.283914    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:12.069528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:12.069495    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:12.215629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:12.215590    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:12.607671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:12.585603    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:12.736396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:12.736359    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:13.136194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:13.136160    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:13.289039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:13.289003    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:13.593155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:13.586762    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:50:14.265341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.265256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:14.406586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.406550    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:14.918417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.918383    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"]
Nov 02 15:50:14.922951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.922933    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:14.941863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.941840    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:50:14.946250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.946223    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"]
Nov 02 15:50:14.954959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.954937    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:50:14.998612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.998571    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqwff\" (UniqueName: \"kubernetes.io/projected/4f7eab21-b124-47a5-8f00-52654ddb743c-kube-api-access-mqwff\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:14.998855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.998779    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:14.999043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:14.999020    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.133147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.133097    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:15.135831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.135798    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.136002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.135947    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqwff\" (UniqueName: \"kubernetes.io/projected/4f7eab21-b124-47a5-8f00-52654ddb743c-kube-api-access-mqwff\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.136092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.136074    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.141671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.138740    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.149093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.149062    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.163031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.163007    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqwff\" (UniqueName: \"kubernetes.io/projected/4f7eab21-b124-47a5-8f00-52654ddb743c-kube-api-access-mqwff\") pod \"maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.237505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.237415    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:15.590728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:15.590525    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"]
Nov 02 15:50:15.599059 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:50:15.599026    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7eab21_b124_47a5_8f00_52654ddb743c.slice/crio-3a82925670a0bc74332aba3e29a7b009ef7ea5b0342a9d92faf892b6d453a85f WatchSource:0}: Error finding container 3a82925670a0bc74332aba3e29a7b009ef7ea5b0342a9d92faf892b6d453a85f: Status 404 returned error can't find the container with id 3a82925670a0bc74332aba3e29a7b009ef7ea5b0342a9d92faf892b6d453a85f
Nov 02 15:50:16.215885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:16.215845    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x" event={"ID":"4f7eab21-b124-47a5-8f00-52654ddb743c","Type":"ContainerStarted","Data":"64a8d5dfcd04413bf1cfd2cb85883f72f3faa61c29ea568df502823f8a5f2750"}
Nov 02 15:50:16.216087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:16.215913    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x" event={"ID":"4f7eab21-b124-47a5-8f00-52654ddb743c","Type":"ContainerStarted","Data":"3a82925670a0bc74332aba3e29a7b009ef7ea5b0342a9d92faf892b6d453a85f"}
Nov 02 15:50:16.259738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:16.259629    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x" podStartSLOduration=2.259615327 podStartE2EDuration="2.259615327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:16.257768269 +0000 UTC m=+3210.511721893" watchObservedRunningTime="2026-11-02 15:50:16.259615327 +0000 UTC m=+3210.513568952"
Nov 02 15:50:16.884805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:16.884715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:17.225741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:17.225704    7019 generic.go:356] "Generic (PLEG): container finished" podID="4f7eab21-b124-47a5-8f00-52654ddb743c" containerID="64a8d5dfcd04413bf1cfd2cb85883f72f3faa61c29ea568df502823f8a5f2750" exitCode=0
Nov 02 15:50:17.225930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:17.225768    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x" event={"ID":"4f7eab21-b124-47a5-8f00-52654ddb743c","Type":"ContainerDied","Data":"64a8d5dfcd04413bf1cfd2cb85883f72f3faa61c29ea568df502823f8a5f2750"}
Nov 02 15:50:18.440966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.440939    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:18.526451 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.526366    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-openshift-ca\") pod \"4f7eab21-b124-47a5-8f00-52654ddb743c\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") "
Nov 02 15:50:18.526605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.526530    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-xks-ca\") pod \"4f7eab21-b124-47a5-8f00-52654ddb743c\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") "
Nov 02 15:50:18.526662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.526630    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7eab21-b124-47a5-8f00-52654ddb743c-kube-api-access-mqwff\" (UniqueName: \"kubernetes.io/projected/4f7eab21-b124-47a5-8f00-52654ddb743c-kube-api-access-mqwff\") pod \"4f7eab21-b124-47a5-8f00-52654ddb743c\" (UID: \"4f7eab21-b124-47a5-8f00-52654ddb743c\") "
Nov 02 15:50:18.527972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.527944    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-openshift-ca" pod "4f7eab21-b124-47a5-8f00-52654ddb743c" (UID: "4f7eab21-b124-47a5-8f00-52654ddb743c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:50:18.529754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.529722    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-xks-ca" pod "4f7eab21-b124-47a5-8f00-52654ddb743c" (UID: "4f7eab21-b124-47a5-8f00-52654ddb743c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:50:18.532131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.532106    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7eab21-b124-47a5-8f00-52654ddb743c-kube-api-access-mqwff" pod "4f7eab21-b124-47a5-8f00-52654ddb743c" (UID: "4f7eab21-b124-47a5-8f00-52654ddb743c"). InnerVolumeSpecName "kube-api-access-mqwff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:50:18.629225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.629189    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:18.629225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.629221    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqwff\" (UniqueName: \"kubernetes.io/projected/4f7eab21-b124-47a5-8f00-52654ddb743c-kube-api-access-mqwff\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:18.629225 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:18.629232    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7eab21-b124-47a5-8f00-52654ddb743c-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:50:19.140839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:19.140800    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:19.250663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:19.247618    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x" event={"ID":"4f7eab21-b124-47a5-8f00-52654ddb743c","Type":"ContainerDied","Data":"3a82925670a0bc74332aba3e29a7b009ef7ea5b0342a9d92faf892b6d453a85f"}
Nov 02 15:50:19.250663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:19.247716    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a82925670a0bc74332aba3e29a7b009ef7ea5b0342a9d92faf892b6d453a85f"
Nov 02 15:50:19.250663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:19.247787    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"
Nov 02 15:50:20.701411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:20.701308    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:22.402050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:22.401877    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:22.711776 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:50:22.711683    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:50:23.994755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:23.994708    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:24.306521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.306418    7019 generic.go:356] "Generic (PLEG): container finished" podID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerID="f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b" exitCode=0
Nov 02 15:50:24.306521 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.306488    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" event={"ID":"47b3fdc1-62fc-438e-aebe-c77c3b960f10","Type":"ContainerDied","Data":"f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b"}
Nov 02 15:50:24.306976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.306960    7019 scope.go:122] "RemoveContainer" containerID="f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b"
Nov 02 15:50:24.309190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.309153    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:50:24.309298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.309248    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:50:24.309348 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.309295    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:50:24.498669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.498342    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:24.646829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:24.646792    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:25.332519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:25.332475    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" event={"ID":"47b3fdc1-62fc-438e-aebe-c77c3b960f10","Type":"ContainerStarted","Data":"d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2"}
Nov 02 15:50:25.812107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:25.812071    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:50:25.813574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:25.813541    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 02 15:50:25.813729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:25.813596    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 02 15:50:26.098001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:26.096538    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:26.140700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:26.140643    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:50:26.812536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:26.812500    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 02 15:50:26.813037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:26.812552    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 02 15:50:27.653003 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:27.652961    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:27.812626 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:27.812586    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 02 15:50:27.813141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:27.812769    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 02 15:50:28.812542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:28.812504    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 02 15:50:28.812788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:28.812567    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 02 15:50:29.230669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:29.230622    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:29.812530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:29.812492    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 02 15:50:29.812758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:29.812552    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 02 15:50:30.817372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:30.817340    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:50:30.818442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:30.818423    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:50:31.289881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:31.289841    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:32.035531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:32.035482    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"]
Nov 02 15:50:32.048091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:32.048063    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0adb49cc-w9sb9"]
Nov 02 15:50:32.879223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:32.871560    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:33.862397 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:33.862366    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c674914e-3f08-4722-a12c-1ac77408927a" path="/var/lib/kubelet/pods/c674914e-3f08-4722-a12c-1ac77408927a/volumes"
Nov 02 15:50:34.441953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:34.441903    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:36.000015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:35.999981    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:37.042911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:37.042879    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"]
Nov 02 15:50:37.055039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:37.055009    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e57edf19-bw7vn"]
Nov 02 15:50:37.861771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:37.861732    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78062598-34cb-4f28-a80e-a669c553b28f" path="/var/lib/kubelet/pods/78062598-34cb-4f28-a80e-a669c553b28f/volumes"
Nov 02 15:50:38.144384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:38.144291    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:40.353287 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:40.353242    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:42.618414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:42.618371    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:42.789761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:42.789727    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:43.061400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:43.061314    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"]
Nov 02 15:50:43.064230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:43.064201    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df27c9e1-5h8ft"]
Nov 02 15:50:43.862283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:43.862247    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912f5584-3ecc-49ce-809a-eb836e0d08a2" path="/var/lib/kubelet/pods/912f5584-3ecc-49ce-809a-eb836e0d08a2/volumes"
Nov 02 15:50:44.528467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:44.528376    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:46.090961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:46.090923    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:47.703513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:47.703472    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:48.035542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:48.035457    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"]
Nov 02 15:50:48.047960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:48.047925    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d776d7b-nh2nr"]
Nov 02 15:50:49.103878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.103835    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:49.592633 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.592594    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:49.695687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.695636    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"]
Nov 02 15:50:49.706301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.706272    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-0a7163-mmtdh"]
Nov 02 15:50:49.741467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.741438    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:49.894337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.894303    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:49.900724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.899603    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f" path="/var/lib/kubelet/pods/4c4a7715-c2aa-4c2f-bf32-44f6227b7d3f/volumes"
Nov 02 15:50:49.900724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:49.900315    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525a417a-3500-453e-ab80-92aea18e3468" path="/var/lib/kubelet/pods/525a417a-3500-453e-ab80-92aea18e3468/volumes"
Nov 02 15:50:50.183080 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:50.182994    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:50.669364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:50.669329    7019 scope.go:122] "RemoveContainer" containerID="67d4f1f119112401d6524a5909f5091904e11842c978ed50133fbfbf322a7de1"
Nov 02 15:50:50.711127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:50.711100    7019 scope.go:122] "RemoveContainer" containerID="e922f00ce13e0ba69cbd270bbe9850c837ca3e00e09529329bcbde470680b13f"
Nov 02 15:50:50.790951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:50.786375    7019 scope.go:122] "RemoveContainer" containerID="628a8ab5d8786f5ac8c6adaab3a1004efaea115516829bb1d6d0337ed9b95393"
Nov 02 15:50:51.523285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:51.523245    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:52.706912 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:50:52.706873    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:50:53.109312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:53.109215    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:54.610880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:54.610839    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:55.699261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:55.699224    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:56.000707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.000603    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:56.180805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.180767    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:56.323255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.323162    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:56.615401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.615349    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:56.732725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.732689    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"]
Nov 02 15:50:56.742659 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.742623    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a55a55-m688x"]
Nov 02 15:50:56.754802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.754776    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:56.922693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:56.922564    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:57.207098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:57.207006    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:57.640705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:57.640667    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:57.794448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:57.794406    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:57.863512 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:57.863474    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7eab21-b124-47a5-8f00-52654ddb743c" path="/var/lib/kubelet/pods/4f7eab21-b124-47a5-8f00-52654ddb743c/volumes"
Nov 02 15:50:57.935982 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:57.935885    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"]
Nov 02 15:50:57.936166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:57.936145    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" containerID="cri-o://257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af" gracePeriod=30
Nov 02 15:50:57.999987 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:57.999948    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:58.062671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.060769    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"]
Nov 02 15:50:58.095705 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.088098    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99a4c14d-z6zf6"]
Nov 02 15:50:58.225270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.225170    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:58.438527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.431532    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:58.568718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.556716    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"]
Nov 02 15:50:58.568718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.566729    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4f7eab21-b124-47a5-8f00-52654ddb743c" containerName="revoke-keys"
Nov 02 15:50:58.568718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.566753    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7eab21-b124-47a5-8f00-52654ddb743c" containerName="revoke-keys"
Nov 02 15:50:58.568718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.566908    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7eab21-b124-47a5-8f00-52654ddb743c" containerName="revoke-keys"
Nov 02 15:50:58.600392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.600231    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:58.605466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.605439    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"]
Nov 02 15:50:58.605803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.605785    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.653120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.653042    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0039c\"/\"default-dockercfg-tbghj\""
Nov 02 15:50:58.653292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.653181    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0039c\"/\"kube-root-ca.crt\""
Nov 02 15:50:58.653292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.653242    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f0039c\"/\"openshift-service-ca.crt\""
Nov 02 15:50:58.653372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.653334    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f0039c\"/\"ipp-route-f0039c-kserve-self-signed-certs\""
Nov 02 15:50:58.659716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.659110    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tmp-dir\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.659716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.659270    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-model-cache\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.659716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.659357    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-home\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.659716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.659565    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-dshm\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.659716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.659635    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45dlk\" (UniqueName: \"kubernetes.io/projected/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-kube-api-access-45dlk\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.660156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.659769    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tls-certs\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.683123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.683087    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"]
Nov 02 15:50:58.687437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.687407    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.728280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.724904    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"]
Nov 02 15:50:58.758283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.758245    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.759759    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.761939    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tmp-dir\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.762015    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-model-cache\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.762085    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-home\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.762335    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zszxl\" (UniqueName: \"kubernetes.io/projected/b0a74621-ad4e-4585-b658-10bfde829ba1-kube-api-access-zszxl\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.762614    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.762925    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-dshm\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.763459    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45dlk\" (UniqueName: \"kubernetes.io/projected/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-kube-api-access-45dlk\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.766670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.763975    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tls-certs\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.844763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.836617    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-home\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.858753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.854429    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-dshm\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.858753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.764439    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.882347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.882311    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-model-cache\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.883773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.883745    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tmp-dir\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.890751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.890725    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tls-certs\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.912499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.912445    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45dlk\" (UniqueName: \"kubernetes.io/projected/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-kube-api-access-45dlk\") pod \"ipp-route-f0039c-kserve-6b66d8db85-gnkzx\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") " pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.936506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.936471    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:58.937026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.936998    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:50:58.961613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.961577    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.961894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.961835    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zszxl\" (UniqueName: \"kubernetes.io/projected/b0a74621-ad4e-4585-b658-10bfde829ba1-kube-api-access-zszxl\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.961894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.961873    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.964407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.964368    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.973113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.973082    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:58.996555 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:58.996522    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zszxl\" (UniqueName: \"kubernetes.io/projected/b0a74621-ad4e-4585-b658-10bfde829ba1-kube-api-access-zszxl\") pod \"maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:59.192970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.192937    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:50:59.275403 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.272830    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:50:59.314666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.314163    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"]
Nov 02 15:50:59.331555 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:50:59.331461    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc772fc1_9d2d_4177_b91b_a6ebdf72b920.slice/crio-19d55d463d392075eb01b3b483b3f3e74a52d19ee8b6b5d9f0046fc2d2458c65 WatchSource:0}: Error finding container 19d55d463d392075eb01b3b483b3f3e74a52d19ee8b6b5d9f0046fc2d2458c65: Status 404 returned error can't find the container with id 19d55d463d392075eb01b3b483b3f3e74a52d19ee8b6b5d9f0046fc2d2458c65
Nov 02 15:50:59.346342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.344733    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:50:59.346342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.344796    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:50:59.512063 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:50:59.512028    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a74621_ad4e_4585_b658_10bfde829ba1.slice/crio-e533a1d2d5d3e2112cfc9b3407afd549e34fc1cb6bfa9f47167d42196f0f6d5c WatchSource:0}: Error finding container e533a1d2d5d3e2112cfc9b3407afd549e34fc1cb6bfa9f47167d42196f0f6d5c: Status 404 returned error can't find the container with id e533a1d2d5d3e2112cfc9b3407afd549e34fc1cb6bfa9f47167d42196f0f6d5c
Nov 02 15:50:59.512221 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.512196    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"]
Nov 02 15:50:59.681000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.680965    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n" event={"ID":"b0a74621-ad4e-4585-b658-10bfde829ba1","Type":"ContainerStarted","Data":"e533a1d2d5d3e2112cfc9b3407afd549e34fc1cb6bfa9f47167d42196f0f6d5c"}
Nov 02 15:50:59.682539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.682508    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx" event={"ID":"cc772fc1-9d2d-4177-b91b-a6ebdf72b920","Type":"ContainerStarted","Data":"d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059"}
Nov 02 15:50:59.682685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.682548    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx" event={"ID":"cc772fc1-9d2d-4177-b91b-a6ebdf72b920","Type":"ContainerStarted","Data":"19d55d463d392075eb01b3b483b3f3e74a52d19ee8b6b5d9f0046fc2d2458c65"}
Nov 02 15:50:59.744322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.744265    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx" podStartSLOduration=1.744250397 podStartE2EDuration="1.744250397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:50:59.742067852 +0000 UTC m=+3253.996021475" watchObservedRunningTime="2026-11-02 15:50:59.744250397 +0000 UTC m=+3253.998204020"
Nov 02 15:50:59.863550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.863512    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e" path="/var/lib/kubelet/pods/ec0b45ea-4edb-42e6-95e3-cfb157c0ba3e/volumes"
Nov 02 15:50:59.937578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:50:59.937537    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:51:00.692803 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:00.692710    7019 generic.go:356] "Generic (PLEG): container finished" podID="b0a74621-ad4e-4585-b658-10bfde829ba1" containerID="9d4efd8d90f476978d3e54d53a6aa279365653cca8b0d7ff19be642f57c5bea2" exitCode=0
Nov 02 15:51:00.692996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:00.692796    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n" event={"ID":"b0a74621-ad4e-4585-b658-10bfde829ba1","Type":"ContainerDied","Data":"9d4efd8d90f476978d3e54d53a6aa279365653cca8b0d7ff19be642f57c5bea2"}
Nov 02 15:51:01.108510 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:01.108406    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:01.856539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:01.856510    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:51:02.013116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.013022    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a74621-ad4e-4585-b658-10bfde829ba1-kube-api-access-zszxl\" (UniqueName: \"kubernetes.io/projected/b0a74621-ad4e-4585-b658-10bfde829ba1-kube-api-access-zszxl\") pod \"b0a74621-ad4e-4585-b658-10bfde829ba1\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") "
Nov 02 15:51:02.013116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.013074    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-openshift-ca\") pod \"b0a74621-ad4e-4585-b658-10bfde829ba1\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") "
Nov 02 15:51:02.013404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.013364    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-xks-ca\") pod \"b0a74621-ad4e-4585-b658-10bfde829ba1\" (UID: \"b0a74621-ad4e-4585-b658-10bfde829ba1\") "
Nov 02 15:51:02.013559 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.013527    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-openshift-ca" pod "b0a74621-ad4e-4585-b658-10bfde829ba1" (UID: "b0a74621-ad4e-4585-b658-10bfde829ba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:02.014144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.014125    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:02.015558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.015534    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a74621-ad4e-4585-b658-10bfde829ba1-kube-api-access-zszxl" pod "b0a74621-ad4e-4585-b658-10bfde829ba1" (UID: "b0a74621-ad4e-4585-b658-10bfde829ba1"). InnerVolumeSpecName "kube-api-access-zszxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:02.015779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.015760    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-xks-ca" pod "b0a74621-ad4e-4585-b658-10bfde829ba1" (UID: "b0a74621-ad4e-4585-b658-10bfde829ba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:02.116030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.115987    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a74621-ad4e-4585-b658-10bfde829ba1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:02.116030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.116021    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zszxl\" (UniqueName: \"kubernetes.io/projected/b0a74621-ad4e-4585-b658-10bfde829ba1-kube-api-access-zszxl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:02.712702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.712668    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"
Nov 02 15:51:02.712702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.712679    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n" event={"ID":"b0a74621-ad4e-4585-b658-10bfde829ba1","Type":"ContainerDied","Data":"e533a1d2d5d3e2112cfc9b3407afd549e34fc1cb6bfa9f47167d42196f0f6d5c"}
Nov 02 15:51:02.712996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.712740    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e533a1d2d5d3e2112cfc9b3407afd549e34fc1cb6bfa9f47167d42196f0f6d5c"
Nov 02 15:51:02.743634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:02.743603    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:03.221898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.221830    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:51:03.333605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333572    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-data\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.333790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333618    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istiod-ca-cert\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.333790 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333687    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-kube-api-access-rrckf\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-kube-api-access-rrckf\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.333938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333795    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-token\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-token\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.333938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333886    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-credential-socket\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.333938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333925    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-podinfo\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.334099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333927    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-data" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:03.334099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.333959    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-envoy\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.334208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.334113    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-socket\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.334208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.334119    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istiod-ca-cert" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:03.334208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.334202    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-certs\") pod \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\" (UID: \"9e68d4b4-3616-40af-86bc-d419e0d0fc0e\") "
Nov 02 15:51:03.334427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.334385    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-credential-socket" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:03.334583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.334472    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-socket" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:03.334677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.334577    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-certs" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:03.335592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.335562    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.335592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.335588    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.335785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.335601    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.335785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.335615    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.335785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.335627    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.336419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.336397    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-kube-api-access-rrckf" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "kube-api-access-rrckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:03.336665 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.336628    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-podinfo" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:51:03.336812 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.336793    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-envoy" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:03.339773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.339750    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-token" pod "9e68d4b4-3616-40af-86bc-d419e0d0fc0e" (UID: "9e68d4b4-3616-40af-86bc-d419e0d0fc0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:03.421999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.421946    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"]
Nov 02 15:51:03.437374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.437323    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrckf\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-kube-api-access-rrckf\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.437374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.437364    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.437374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.437378    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.437721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.437387    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e68d4b4-3616-40af-86bc-d419e0d0fc0e-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:03.547056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.546967    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:03.721770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.721731    7019 generic.go:356] "Generic (PLEG): container finished" podID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerID="257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af" exitCode=0
Nov 02 15:51:03.721964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.721802    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"
Nov 02 15:51:03.721964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.721810    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" event={"ID":"9e68d4b4-3616-40af-86bc-d419e0d0fc0e","Type":"ContainerDied","Data":"257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af"}
Nov 02 15:51:03.721964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.721897    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78" event={"ID":"9e68d4b4-3616-40af-86bc-d419e0d0fc0e","Type":"ContainerDied","Data":"a4edfd3db32a3a871cb0a98caad77cefc2743efcf9324b73106bd5620185484b"}
Nov 02 15:51:03.721964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.721930    7019 scope.go:122] "RemoveContainer" containerID="257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af"
Nov 02 15:51:03.733730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.733712    7019 scope.go:122] "RemoveContainer" containerID="90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243"
Nov 02 15:51:03.744546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.744525    7019 scope.go:122] "RemoveContainer" containerID="257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af"
Nov 02 15:51:03.744848 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:03.744826    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af\": container with ID starting with 257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af not found: ID does not exist" containerID="257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af"
Nov 02 15:51:03.744927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.744856    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af"} err="failed to get container status \"257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af\": rpc error: code = NotFound desc = could not find container \"257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af\": container with ID starting with 257239fbd37f2b0d2b9a1b5ca9f2305ca05980dae0b0ce33ce6d7921628903af not found: ID does not exist"
Nov 02 15:51:03.744927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.744873    7019 scope.go:122] "RemoveContainer" containerID="90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243"
Nov 02 15:51:03.745102 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:03.745088    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243\": container with ID starting with 90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243 not found: ID does not exist" containerID="90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243"
Nov 02 15:51:03.745143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.745105    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243"} err="failed to get container status \"90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243\": rpc error: code = NotFound desc = could not find container \"90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243\": container with ID starting with 90023dad9dad9c31f38802fa0804aed455a62e6e0148055eb0cce9fa1a54a243 not found: ID does not exist"
Nov 02 15:51:03.778703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.777231    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"]
Nov 02 15:51:03.790364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.790333    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0a7163-openshift-default-568b79c699-hhv78"]
Nov 02 15:51:03.863172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.863141    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" path="/var/lib/kubelet/pods/9e68d4b4-3616-40af-86bc-d419e0d0fc0e/volumes"
Nov 02 15:51:03.893447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:03.893395    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:04.040043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.040007    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:04.181727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.181202    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"]
Nov 02 15:51:04.181727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.181440    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy" containerID="cri-o://b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1" gracePeriod=30
Nov 02 15:51:04.246508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.246473    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:04.740200 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740167    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"]
Nov 02 15:51:04.740781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740759    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b0a74621-ad4e-4585-b658-10bfde829ba1" containerName="revoke-keys"
Nov 02 15:51:04.740781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740774    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a74621-ad4e-4585-b658-10bfde829ba1" containerName="revoke-keys"
Nov 02 15:51:04.741259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740815    7019 cpu_manager.go:414] "containerMap: removing container" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy"
Nov 02 15:51:04.741259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740821    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy"
Nov 02 15:51:04.741259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740840    7019 cpu_manager.go:414] "containerMap: removing container" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy"
Nov 02 15:51:04.741259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740844    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy"
Nov 02 15:51:04.741259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740931    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy"
Nov 02 15:51:04.741259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.740943    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a74621-ad4e-4585-b658-10bfde829ba1" containerName="revoke-keys"
Nov 02 15:51:04.744272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.744255    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.763199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.763176    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"]
Nov 02 15:51:04.772688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.772190    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:51:04.772688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.772443    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:51:04.859238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.859198    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzjq2\" (UniqueName: \"kubernetes.io/projected/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-kube-api-access-pzjq2\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.859441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.859401    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.859499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.859440    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.886534 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.886487    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:04.960862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.960821    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.961062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.960888    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.961062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.961006    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzjq2\" (UniqueName: \"kubernetes.io/projected/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-kube-api-access-pzjq2\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.961642 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.961619    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.963318 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.963299    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:04.988664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:04.988507    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzjq2\" (UniqueName: \"kubernetes.io/projected/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-kube-api-access-pzjq2\") pod \"maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:05.057934 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:05.057863    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:05.379198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:05.379170    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"]
Nov 02 15:51:05.379552 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:51:05.379207    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a8ce2c_95d0_4fe0_9f94_98e04d45f15a.slice/crio-312f24807b1428d3f031a33a13f27d915ad9e949b740caba5fcc2222a96ecbe5 WatchSource:0}: Error finding container 312f24807b1428d3f031a33a13f27d915ad9e949b740caba5fcc2222a96ecbe5: Status 404 returned error can't find the container with id 312f24807b1428d3f031a33a13f27d915ad9e949b740caba5fcc2222a96ecbe5
Nov 02 15:51:05.740027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:05.739941    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl" event={"ID":"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a","Type":"ContainerStarted","Data":"312f24807b1428d3f031a33a13f27d915ad9e949b740caba5fcc2222a96ecbe5"}
Nov 02 15:51:06.628731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:06.597570    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:06.760719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:06.760683    7019 generic.go:356] "Generic (PLEG): container finished" podID="33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" containerID="4e2739f10446d2ce20de81ab88ee6136f2c5f149d2ca9c1b677a3c0d55d86898" exitCode=0
Nov 02 15:51:06.760904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:06.760768    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl" event={"ID":"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a","Type":"ContainerDied","Data":"4e2739f10446d2ce20de81ab88ee6136f2c5f149d2ca9c1b677a3c0d55d86898"}
Nov 02 15:51:07.932032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:07.932006    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:08.006560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.006524    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-xks-ca\") pod \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") "
Nov 02 15:51:08.006783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.006583    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-kube-api-access-pzjq2\" (UniqueName: \"kubernetes.io/projected/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-kube-api-access-pzjq2\") pod \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") "
Nov 02 15:51:08.006875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.006854    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-openshift-ca\") pod \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\" (UID: \"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a\") "
Nov 02 15:51:08.007717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.007341    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-openshift-ca" pod "33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" (UID: "33a8ce2c-95d0-4fe0-9f94-98e04d45f15a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:08.007717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.007578    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:08.017007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.016976    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-xks-ca" pod "33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" (UID: "33a8ce2c-95d0-4fe0-9f94-98e04d45f15a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:08.017198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.017166    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-kube-api-access-pzjq2" pod "33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" (UID: "33a8ce2c-95d0-4fe0-9f94-98e04d45f15a"). InnerVolumeSpecName "kube-api-access-pzjq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:08.108743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.108608    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:08.108743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.108671    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzjq2\" (UniqueName: \"kubernetes.io/projected/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a-kube-api-access-pzjq2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:08.164891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.164853    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:08.782243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.782213    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"
Nov 02 15:51:08.782454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.782237    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl" event={"ID":"33a8ce2c-95d0-4fe0-9f94-98e04d45f15a","Type":"ContainerDied","Data":"312f24807b1428d3f031a33a13f27d915ad9e949b740caba5fcc2222a96ecbe5"}
Nov 02 15:51:08.782454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.782298    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="312f24807b1428d3f031a33a13f27d915ad9e949b740caba5fcc2222a96ecbe5"
Nov 02 15:51:08.937830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.937794    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:51:08.951509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:08.951481    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:51:09.234216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.234187    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:09.384676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.384472    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:09.513616 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.513592    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:51:09.523121 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.523095    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:09.638394 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638361    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-certs\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638401    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-envoy\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638496    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-socket\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638524    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-data\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638589    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-kube-api-access-xm26c\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-kube-api-access-xm26c\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638638    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02302e75-aacc-4653-b806-4f06991d24c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02302e75-aacc-4653-b806-4f06991d24c3-istiod-ca-cert\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638711    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02302e75-aacc-4653-b806-4f06991d24c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02302e75-aacc-4653-b806-4f06991d24c3-istio-podinfo\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638730    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-certs" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:09.638881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638759    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-istio-token\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-istio-token\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.638881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638868    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-data" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:09.639038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638917    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-credential-socket\") pod \"02302e75-aacc-4653-b806-4f06991d24c3\" (UID: \"02302e75-aacc-4653-b806-4f06991d24c3\") "
Nov 02 15:51:09.639038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.638944    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-socket" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:09.639230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.639208    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-credential-socket" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:09.639324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.639250    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02302e75-aacc-4653-b806-4f06991d24c3-istiod-ca-cert" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:09.639628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.639604    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.639778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.639632    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.639778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.639660    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02302e75-aacc-4653-b806-4f06991d24c3-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.639778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.639675    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.639778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.639690    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.640994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.640967    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-envoy" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:09.641097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.641072    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-istio-token" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:09.641144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.641108    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-kube-api-access-xm26c" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "kube-api-access-xm26c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:09.641290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.641270    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02302e75-aacc-4653-b806-4f06991d24c3-istio-podinfo" pod "02302e75-aacc-4653-b806-4f06991d24c3" (UID: "02302e75-aacc-4653-b806-4f06991d24c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:51:09.741564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.741480    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02302e75-aacc-4653-b806-4f06991d24c3-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.741564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.741507    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm26c\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-kube-api-access-xm26c\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.741564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.741518    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02302e75-aacc-4653-b806-4f06991d24c3-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.741564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.741526    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02302e75-aacc-4653-b806-4f06991d24c3-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:09.792147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.792113    7019 generic.go:356] "Generic (PLEG): container finished" podID="02302e75-aacc-4653-b806-4f06991d24c3" containerID="b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1" exitCode=0
Nov 02 15:51:09.792324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.792179    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"
Nov 02 15:51:09.792324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.792198    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" event={"ID":"02302e75-aacc-4653-b806-4f06991d24c3","Type":"ContainerDied","Data":"b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1"}
Nov 02 15:51:09.792324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.792267    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7" event={"ID":"02302e75-aacc-4653-b806-4f06991d24c3","Type":"ContainerDied","Data":"191610b24a51bf60e1910e1c30f4e4c214ec52ce76c59d8d06d56b52cbe2c10a"}
Nov 02 15:51:09.792324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.792295    7019 scope.go:122] "RemoveContainer" containerID="b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1"
Nov 02 15:51:09.814081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.813972    7019 scope.go:122] "RemoveContainer" containerID="c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d"
Nov 02 15:51:09.817686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.817616    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:51:09.879848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.877912    7019 scope.go:122] "RemoveContainer" containerID="b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1"
Nov 02 15:51:09.880550 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:09.880410    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1\": container with ID starting with b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1 not found: ID does not exist" containerID="b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1"
Nov 02 15:51:09.880550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.880454    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1"} err="failed to get container status \"b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1\": rpc error: code = NotFound desc = could not find container \"b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1\": container with ID starting with b7dc5fc1750155f8c37542e57e04ecf8586008384fde98ccf5a17d7228e60ee1 not found: ID does not exist"
Nov 02 15:51:09.880550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.880477    7019 scope.go:122] "RemoveContainer" containerID="c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d"
Nov 02 15:51:09.886688 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:09.886642    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d\": container with ID starting with c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d not found: ID does not exist" containerID="c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d"
Nov 02 15:51:09.886860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.886831    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d"} err="failed to get container status \"c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d\": rpc error: code = NotFound desc = could not find container \"c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d\": container with ID starting with c1e37266e693614541634a1c40f8982739d0275fad6b1255aeb7a3c832f9117d not found: ID does not exist"
Nov 02 15:51:09.917125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.915709    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"]
Nov 02 15:51:09.987669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:09.987182    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a55a55-openshift-default-786b6b46ff-dh9m7"]
Nov 02 15:51:10.072991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.072897    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"]
Nov 02 15:51:10.073505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073491    7019 cpu_manager.go:414] "containerMap: removing container" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy"
Nov 02 15:51:10.073505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073505    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy"
Nov 02 15:51:10.073676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073516    7019 cpu_manager.go:414] "containerMap: removing container" podUID="33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" containerName="revoke-keys"
Nov 02 15:51:10.073676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073522    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" containerName="revoke-keys"
Nov 02 15:51:10.073676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073634    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy"
Nov 02 15:51:10.073676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073663    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy"
Nov 02 15:51:10.073676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073671    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" containerName="revoke-keys"
Nov 02 15:51:10.073676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.073677    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e68d4b4-3616-40af-86bc-d419e0d0fc0e" containerName="istio-proxy"
Nov 02 15:51:10.077262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.077239    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.100970 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.100939    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:51:10.110491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.102210    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:51:10.110491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.103437    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"]
Nov 02 15:51:10.116218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.116189    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"]
Nov 02 15:51:10.117168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.117147    7019 cpu_manager.go:414] "containerMap: removing container" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy"
Nov 02 15:51:10.117231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.117172    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02302e75-aacc-4653-b806-4f06991d24c3" containerName="istio-proxy"
Nov 02 15:51:10.120463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.120443    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.147732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.147699    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"]
Nov 02 15:51:10.150679 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.150620    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.150813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.150757    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.150813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.150788    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r482w\" (UniqueName: \"kubernetes.io/projected/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-kube-api-access-r482w\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.252538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.252500    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334660bb-1214-473e-b530-84d56a927b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.252538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.252541    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334660bb-1214-473e-b530-84d56a927b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.252815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.252692    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc784\" (UniqueName: \"kubernetes.io/projected/334660bb-1214-473e-b530-84d56a927b42-kube-api-access-rc784\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.252815 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.252742    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.252884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.252849    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.252884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.252872    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r482w\" (UniqueName: \"kubernetes.io/projected/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-kube-api-access-r482w\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.253412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.253381    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.255238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.255211    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.277947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.277918    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r482w\" (UniqueName: \"kubernetes.io/projected/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-kube-api-access-r482w\") pod \"maas-api-revoke-sub-e4e2d6b4-4mctf\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.354131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.354054    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334660bb-1214-473e-b530-84d56a927b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.354131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.354091    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334660bb-1214-473e-b530-84d56a927b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.354332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.354156    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc784\" (UniqueName: \"kubernetes.io/projected/334660bb-1214-473e-b530-84d56a927b42-kube-api-access-rc784\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.354822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.354799    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334660bb-1214-473e-b530-84d56a927b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.356429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.356406    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334660bb-1214-473e-b530-84d56a927b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.387026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.386992    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc784\" (UniqueName: \"kubernetes.io/projected/334660bb-1214-473e-b530-84d56a927b42-kube-api-access-rc784\") pod \"maas-api-revoke-sub-63ecad19-6fzsk\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.392740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.392701    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:10.441757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.441724    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:10.621917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.621584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:10.707864 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:51:10.707827    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbf4c2d_fecb_4bf2_9c87_71ab236580f0.slice/crio-52a643addc38a32094b8d661510c3f5d2e2e540a394d283bc8785d3e38e1e52d WatchSource:0}: Error finding container 52a643addc38a32094b8d661510c3f5d2e2e540a394d283bc8785d3e38e1e52d: Status 404 returned error can't find the container with id 52a643addc38a32094b8d661510c3f5d2e2e540a394d283bc8785d3e38e1e52d
Nov 02 15:51:10.709338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.709313    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"]
Nov 02 15:51:10.783672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.783629    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"]
Nov 02 15:51:10.789106 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:51:10.789076    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334660bb_1214_473e_b530_84d56a927b42.slice/crio-28522d110f656a26bfafa480f32f8affadb7364e2d8a9b6d1ed019fd1c79e4f3 WatchSource:0}: Error finding container 28522d110f656a26bfafa480f32f8affadb7364e2d8a9b6d1ed019fd1c79e4f3: Status 404 returned error can't find the container with id 28522d110f656a26bfafa480f32f8affadb7364e2d8a9b6d1ed019fd1c79e4f3
Nov 02 15:51:10.811366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.811316    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk" event={"ID":"334660bb-1214-473e-b530-84d56a927b42","Type":"ContainerStarted","Data":"28522d110f656a26bfafa480f32f8affadb7364e2d8a9b6d1ed019fd1c79e4f3"}
Nov 02 15:51:10.814338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:10.814312    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf" event={"ID":"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0","Type":"ContainerStarted","Data":"52a643addc38a32094b8d661510c3f5d2e2e540a394d283bc8785d3e38e1e52d"}
Nov 02 15:51:11.823930 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:11.823895    7019 generic.go:356] "Generic (PLEG): container finished" podID="334660bb-1214-473e-b530-84d56a927b42" containerID="7c0284ac9cb9ecebc5348fb0d816f1646c63fb75b3bfa878119d1a5c7ebe86f0" exitCode=0
Nov 02 15:51:11.824385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:11.823977    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk" event={"ID":"334660bb-1214-473e-b530-84d56a927b42","Type":"ContainerDied","Data":"7c0284ac9cb9ecebc5348fb0d816f1646c63fb75b3bfa878119d1a5c7ebe86f0"}
Nov 02 15:51:11.825922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:11.825894    7019 generic.go:356] "Generic (PLEG): container finished" podID="7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" containerID="0572a23feb50afca167fbf989f9224eb107fabd01aad48abc89df556ce34f1fd" exitCode=0
Nov 02 15:51:11.826062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:11.825977    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf" event={"ID":"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0","Type":"ContainerDied","Data":"0572a23feb50afca167fbf989f9224eb107fabd01aad48abc89df556ce34f1fd"}
Nov 02 15:51:11.862452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:11.862414    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02302e75-aacc-4653-b806-4f06991d24c3" path="/var/lib/kubelet/pods/02302e75-aacc-4653-b806-4f06991d24c3/volumes"
Nov 02 15:51:12.200635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:12.200545    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:12.645007 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:12.644961    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:51:13.021055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.021021    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:13.036596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.036572    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:13.113181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.113145    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-openshift-ca\") pod \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") "
Nov 02 15:51:13.113364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.113281    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-kube-api-access-r482w\" (UniqueName: \"kubernetes.io/projected/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-kube-api-access-r482w\") pod \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") "
Nov 02 15:51:13.113410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.113368    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-xks-ca\") pod \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\" (UID: \"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0\") "
Nov 02 15:51:13.113886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.113858    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-openshift-ca" pod "7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" (UID: "7cbf4c2d-fecb-4bf2-9c87-71ab236580f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:13.114222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.114203    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:13.116791 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.116753    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-xks-ca" pod "7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" (UID: "7cbf4c2d-fecb-4bf2-9c87-71ab236580f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:13.116915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.116806    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-kube-api-access-r482w" pod "7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" (UID: "7cbf4c2d-fecb-4bf2-9c87-71ab236580f0"). InnerVolumeSpecName "kube-api-access-r482w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:13.215249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.215169    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334660bb-1214-473e-b530-84d56a927b42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334660bb-1214-473e-b530-84d56a927b42-maas-api-openshift-ca\") pod \"334660bb-1214-473e-b530-84d56a927b42\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") "
Nov 02 15:51:13.215249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.215228    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/334660bb-1214-473e-b530-84d56a927b42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334660bb-1214-473e-b530-84d56a927b42-maas-api-xks-ca\") pod \"334660bb-1214-473e-b530-84d56a927b42\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") "
Nov 02 15:51:13.215505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.215278    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334660bb-1214-473e-b530-84d56a927b42-kube-api-access-rc784\" (UniqueName: \"kubernetes.io/projected/334660bb-1214-473e-b530-84d56a927b42-kube-api-access-rc784\") pod \"334660bb-1214-473e-b530-84d56a927b42\" (UID: \"334660bb-1214-473e-b530-84d56a927b42\") "
Nov 02 15:51:13.215630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.215604    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334660bb-1214-473e-b530-84d56a927b42-maas-api-openshift-ca" pod "334660bb-1214-473e-b530-84d56a927b42" (UID: "334660bb-1214-473e-b530-84d56a927b42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:13.216068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.216043    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r482w\" (UniqueName: \"kubernetes.io/projected/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-kube-api-access-r482w\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:13.216068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.216065    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:13.216068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.216074    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/334660bb-1214-473e-b530-84d56a927b42-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:13.217469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.217441    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334660bb-1214-473e-b530-84d56a927b42-maas-api-xks-ca" pod "334660bb-1214-473e-b530-84d56a927b42" (UID: "334660bb-1214-473e-b530-84d56a927b42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:13.217578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.217466    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334660bb-1214-473e-b530-84d56a927b42-kube-api-access-rc784" pod "334660bb-1214-473e-b530-84d56a927b42" (UID: "334660bb-1214-473e-b530-84d56a927b42"). InnerVolumeSpecName "kube-api-access-rc784". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:13.317489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.317458    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/334660bb-1214-473e-b530-84d56a927b42-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:13.317489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.317486    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc784\" (UniqueName: \"kubernetes.io/projected/334660bb-1214-473e-b530-84d56a927b42-kube-api-access-rc784\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:13.864364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.862743    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"
Nov 02 15:51:13.871688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.869570    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf" event={"ID":"7cbf4c2d-fecb-4bf2-9c87-71ab236580f0","Type":"ContainerDied","Data":"52a643addc38a32094b8d661510c3f5d2e2e540a394d283bc8785d3e38e1e52d"}
Nov 02 15:51:13.871688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.869621    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a643addc38a32094b8d661510c3f5d2e2e540a394d283bc8785d3e38e1e52d"
Nov 02 15:51:13.871688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.871674    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk" event={"ID":"334660bb-1214-473e-b530-84d56a927b42","Type":"ContainerDied","Data":"28522d110f656a26bfafa480f32f8affadb7364e2d8a9b6d1ed019fd1c79e4f3"}
Nov 02 15:51:13.871977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.871719    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28522d110f656a26bfafa480f32f8affadb7364e2d8a9b6d1ed019fd1c79e4f3"
Nov 02 15:51:13.871977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.871793    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"
Nov 02 15:51:13.904356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:13.904316    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:14.331901 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:14.331806    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:15.796672 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:15.795637    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:15.947957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:15.947619    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:16.095730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:16.095411    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:16.386478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:16.386273    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:16.507843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:16.507801    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"]
Nov 02 15:51:16.537569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:16.537515    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5721b1-p8xhl"]
Nov 02 15:51:16.547093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:16.547062    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:16.776770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:16.775478    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:17.139299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.139267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:17.307300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.307258    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"]
Nov 02 15:51:17.308189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.308162    7019 cpu_manager.go:414] "containerMap: removing container" podUID="334660bb-1214-473e-b530-84d56a927b42" containerName="revoke-keys"
Nov 02 15:51:17.308189 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.308187    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334660bb-1214-473e-b530-84d56a927b42" containerName="revoke-keys"
Nov 02 15:51:17.308383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.308258    7019 cpu_manager.go:414] "containerMap: removing container" podUID="7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" containerName="revoke-keys"
Nov 02 15:51:17.308383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.308266    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" containerName="revoke-keys"
Nov 02 15:51:17.308447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.308412    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="334660bb-1214-473e-b530-84d56a927b42" containerName="revoke-keys"
Nov 02 15:51:17.308447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.308436    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" containerName="revoke-keys"
Nov 02 15:51:17.312000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.311973    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.328458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.328428    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"]
Nov 02 15:51:17.412799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.407680    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwkw\" (UniqueName: \"kubernetes.io/projected/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-kube-api-access-bjwkw\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.412799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.407781    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.412799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.407919    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.413150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.413053    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:51:17.413150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.413136    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:51:17.510669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.510455    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.510669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.510601    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.510973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.510776    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwkw\" (UniqueName: \"kubernetes.io/projected/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-kube-api-access-bjwkw\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.519020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.512290    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.519020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.517101    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.535811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.535767    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwkw\" (UniqueName: \"kubernetes.io/projected/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-kube-api-access-bjwkw\") pod \"maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.627937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.627702    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:17.926180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:17.865975    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a8ce2c-95d0-4fe0-9f94-98e04d45f15a" path="/var/lib/kubelet/pods/33a8ce2c-95d0-4fe0-9f94-98e04d45f15a/volumes"
Nov 02 15:51:18.016015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.011403    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"]
Nov 02 15:51:18.452592 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.452452    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main" containerID="cri-o://42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16" gracePeriod=45
Nov 02 15:51:18.827807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.827726    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:51:18.835929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.835865    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d73acb-b8ac-4620-a00e-0b3a5a719560-kube-api-access-gs9lx\" (UniqueName: \"kubernetes.io/projected/95d73acb-b8ac-4620-a00e-0b3a5a719560-kube-api-access-gs9lx\") pod \"95d73acb-b8ac-4620-a00e-0b3a5a719560\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") "
Nov 02 15:51:18.836058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.835995    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-tmp-dir\") pod \"95d73acb-b8ac-4620-a00e-0b3a5a719560\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") "
Nov 02 15:51:18.836101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.836071    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95d73acb-b8ac-4620-a00e-0b3a5a719560-tls-certs\" (UniqueName: \"kubernetes.io/secret/95d73acb-b8ac-4620-a00e-0b3a5a719560-tls-certs\") pod \"95d73acb-b8ac-4620-a00e-0b3a5a719560\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") "
Nov 02 15:51:18.836101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.836092    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-home\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-home\") pod \"95d73acb-b8ac-4620-a00e-0b3a5a719560\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") "
Nov 02 15:51:18.836172 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.836116    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-dshm\") pod \"95d73acb-b8ac-4620-a00e-0b3a5a719560\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") "
Nov 02 15:51:18.836209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.836194    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-model-cache\") pod \"95d73acb-b8ac-4620-a00e-0b3a5a719560\" (UID: \"95d73acb-b8ac-4620-a00e-0b3a5a719560\") "
Nov 02 15:51:18.836913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.836889    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-home" pod "95d73acb-b8ac-4620-a00e-0b3a5a719560" (UID: "95d73acb-b8ac-4620-a00e-0b3a5a719560"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.837132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.837111    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-model-cache" pod "95d73acb-b8ac-4620-a00e-0b3a5a719560" (UID: "95d73acb-b8ac-4620-a00e-0b3a5a719560"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.837230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.837103    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-tmp-dir" pod "95d73acb-b8ac-4620-a00e-0b3a5a719560" (UID: "95d73acb-b8ac-4620-a00e-0b3a5a719560"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.839018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.838981    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-dshm" pod "95d73acb-b8ac-4620-a00e-0b3a5a719560" (UID: "95d73acb-b8ac-4620-a00e-0b3a5a719560"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:51:18.839198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.839174    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d73acb-b8ac-4620-a00e-0b3a5a719560-kube-api-access-gs9lx" pod "95d73acb-b8ac-4620-a00e-0b3a5a719560" (UID: "95d73acb-b8ac-4620-a00e-0b3a5a719560"). InnerVolumeSpecName "kube-api-access-gs9lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:18.839458 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.839428    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95d73acb-b8ac-4620-a00e-0b3a5a719560-tls-certs" pod "95d73acb-b8ac-4620-a00e-0b3a5a719560" (UID: "95d73acb-b8ac-4620-a00e-0b3a5a719560"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:18.938138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.938099    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.938138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.938141    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs9lx\" (UniqueName: \"kubernetes.io/projected/95d73acb-b8ac-4620-a00e-0b3a5a719560-kube-api-access-gs9lx\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.938356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.938158    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.938356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.938192    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95d73acb-b8ac-4620-a00e-0b3a5a719560-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.938356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.938219    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:18.938356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:18.938233    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95d73acb-b8ac-4620-a00e-0b3a5a719560-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:19.022670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.022438    7019 generic.go:356] "Generic (PLEG): container finished" podID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerID="42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16" exitCode=0
Nov 02 15:51:19.022670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.022508    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" event={"ID":"95d73acb-b8ac-4620-a00e-0b3a5a719560","Type":"ContainerDied","Data":"42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16"}
Nov 02 15:51:19.022670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.022558    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7" event={"ID":"95d73acb-b8ac-4620-a00e-0b3a5a719560","Type":"ContainerDied","Data":"1b280fa60af0e124f0834ac2534f5811661c26f181cd0310e12353b20562548e"}
Nov 02 15:51:19.022670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.022577    7019 scope.go:122] "RemoveContainer" containerID="42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16"
Nov 02 15:51:19.023046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.022695    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"
Nov 02 15:51:19.025838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.025791    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp" event={"ID":"70eedf6e-3a0f-4c3a-8743-cde6b687d17f","Type":"ContainerStarted","Data":"d5344835370283329dbac958766144c0423bab884fd39ebf25c5d84e9d9b315d"}
Nov 02 15:51:19.025981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.025861    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp" event={"ID":"70eedf6e-3a0f-4c3a-8743-cde6b687d17f","Type":"ContainerStarted","Data":"24badb6b530ce8a8be793aa3673b163772c4f54fb1243902aaf91a5b8097985e"}
Nov 02 15:51:19.036241 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.036224    7019 scope.go:122] "RemoveContainer" containerID="42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16"
Nov 02 15:51:19.036523 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:19.036504    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16\": container with ID starting with 42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16 not found: ID does not exist" containerID="42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16"
Nov 02 15:51:19.036576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.036533    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16"} err="failed to get container status \"42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16\": rpc error: code = NotFound desc = could not find container \"42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16\": container with ID starting with 42c64df8a3352197aff13531051eb8dd22f9226216230b342c79dff31a0dce16 not found: ID does not exist"
Nov 02 15:51:19.103777 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.103633    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"]
Nov 02 15:51:19.110203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.110170    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-1cc1b2/simulator-w5-1cc1b2-kserve-866696f9fb-ks9m7"]
Nov 02 15:51:19.141453 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.141402    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp" podStartSLOduration=2.141390413 podStartE2EDuration="2.141390413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:19.137809544 +0000 UTC m=+3273.391763173" watchObservedRunningTime="2026-11-02 15:51:19.141390413 +0000 UTC m=+3273.395344037"
Nov 02 15:51:19.230669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.226819    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:19.862661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:19.862626    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" path="/var/lib/kubelet/pods/95d73acb-b8ac-4620-a00e-0b3a5a719560/volumes"
Nov 02 15:51:20.038660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:20.038612    7019 generic.go:356] "Generic (PLEG): container finished" podID="70eedf6e-3a0f-4c3a-8743-cde6b687d17f" containerID="d5344835370283329dbac958766144c0423bab884fd39ebf25c5d84e9d9b315d" exitCode=0
Nov 02 15:51:20.038853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:20.038680    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp" event={"ID":"70eedf6e-3a0f-4c3a-8743-cde6b687d17f","Type":"ContainerDied","Data":"d5344835370283329dbac958766144c0423bab884fd39ebf25c5d84e9d9b315d"}
Nov 02 15:51:20.871838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:20.871800    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:21.230336 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.230307    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:21.270455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.270420    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-openshift-ca\") pod \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") "
Nov 02 15:51:21.270661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.270549    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-kube-api-access-bjwkw\" (UniqueName: \"kubernetes.io/projected/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-kube-api-access-bjwkw\") pod \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") "
Nov 02 15:51:21.270661 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.270588    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-xks-ca\") pod \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\" (UID: \"70eedf6e-3a0f-4c3a-8743-cde6b687d17f\") "
Nov 02 15:51:21.270965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.270933    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-openshift-ca" pod "70eedf6e-3a0f-4c3a-8743-cde6b687d17f" (UID: "70eedf6e-3a0f-4c3a-8743-cde6b687d17f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:21.271122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.271095    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:21.273618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.273593    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-xks-ca" pod "70eedf6e-3a0f-4c3a-8743-cde6b687d17f" (UID: "70eedf6e-3a0f-4c3a-8743-cde6b687d17f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:21.274768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.274267    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-kube-api-access-bjwkw" pod "70eedf6e-3a0f-4c3a-8743-cde6b687d17f" (UID: "70eedf6e-3a0f-4c3a-8743-cde6b687d17f"). InnerVolumeSpecName "kube-api-access-bjwkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:21.372509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.372457    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjwkw\" (UniqueName: \"kubernetes.io/projected/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-kube-api-access-bjwkw\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:21.372509 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:21.372503    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70eedf6e-3a0f-4c3a-8743-cde6b687d17f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:22.058357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:22.058323    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"
Nov 02 15:51:22.058357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:22.058343    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp" event={"ID":"70eedf6e-3a0f-4c3a-8743-cde6b687d17f","Type":"ContainerDied","Data":"24badb6b530ce8a8be793aa3673b163772c4f54fb1243902aaf91a5b8097985e"}
Nov 02 15:51:22.058797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:22.058399    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24badb6b530ce8a8be793aa3673b163772c4f54fb1243902aaf91a5b8097985e"
Nov 02 15:51:22.386881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:22.386584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:22.710225 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:22.710135    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:51:23.226875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:23.226839    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:23.378199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:23.378157    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:23.546048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:23.545940    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:23.992143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:23.992109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:24.046806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.046772    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"]
Nov 02 15:51:24.047421 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.047405    7019 cpu_manager.go:414] "containerMap: removing container" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main"
Nov 02 15:51:24.047480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.047421    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main"
Nov 02 15:51:24.047480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.047447    7019 cpu_manager.go:414] "containerMap: removing container" podUID="70eedf6e-3a0f-4c3a-8743-cde6b687d17f" containerName="revoke-keys"
Nov 02 15:51:24.047480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.047454    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70eedf6e-3a0f-4c3a-8743-cde6b687d17f" containerName="revoke-keys"
Nov 02 15:51:24.047586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.047538    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d73acb-b8ac-4620-a00e-0b3a5a719560" containerName="main"
Nov 02 15:51:24.047586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.047548    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="70eedf6e-3a0f-4c3a-8743-cde6b687d17f" containerName="revoke-keys"
Nov 02 15:51:24.050273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.050257    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.072462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.072433    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"]
Nov 02 15:51:24.090375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.090288    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:51:24.090590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.090417    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:51:24.130422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.130389    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.130578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.130446    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qnj\" (UniqueName: \"kubernetes.io/projected/71e48d42-f103-4187-8d90-4c4beed3c35e-kube-api-access-z5qnj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.130795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.130772    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.231715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.231596    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.231715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.231693    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.232212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.231759    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qnj\" (UniqueName: \"kubernetes.io/projected/71e48d42-f103-4187-8d90-4c4beed3c35e-kube-api-access-z5qnj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.232433 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.232402    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.234065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.234045    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.256335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.256271    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qnj\" (UniqueName: \"kubernetes.io/projected/71e48d42-f103-4187-8d90-4c4beed3c35e-kube-api-access-z5qnj\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.378951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.378918    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:24.684072 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:51:24.684026    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e48d42_f103_4187_8d90_4c4beed3c35e.slice/crio-268c2eb210a844e83744874bf9aef73a19322e90093c904b85f60730ea37964f WatchSource:0}: Error finding container 268c2eb210a844e83744874bf9aef73a19322e90093c904b85f60730ea37964f: Status 404 returned error can't find the container with id 268c2eb210a844e83744874bf9aef73a19322e90093c904b85f60730ea37964f
Nov 02 15:51:24.685016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:24.684992    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"]
Nov 02 15:51:25.088673 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.088622    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn" event={"ID":"71e48d42-f103-4187-8d90-4c4beed3c35e","Type":"ContainerStarted","Data":"d9376ce1f480cf34152700a51c359bc8543fd67a0cf5b66e916988409c745f12"}
Nov 02 15:51:25.088850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.088703    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn" event={"ID":"71e48d42-f103-4187-8d90-4c4beed3c35e","Type":"ContainerStarted","Data":"268c2eb210a844e83744874bf9aef73a19322e90093c904b85f60730ea37964f"}
Nov 02 15:51:25.090429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.090404    7019 generic.go:356] "Generic (PLEG): container finished" podID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerID="e624d8538a0cfe9baaf4e3dac96c76b088720eac9e10c8ce91032d62d5befc34" exitCode=0
Nov 02 15:51:25.090550 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.090453    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" event={"ID":"89d220ea-b8e7-492d-9036-68c13c08ece9","Type":"ContainerDied","Data":"e624d8538a0cfe9baaf4e3dac96c76b088720eac9e10c8ce91032d62d5befc34"}
Nov 02 15:51:25.090810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.090797    7019 scope.go:122] "RemoveContainer" containerID="e624d8538a0cfe9baaf4e3dac96c76b088720eac9e10c8ce91032d62d5befc34"
Nov 02 15:51:25.092342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.092311    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:51:25.092401 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.092382    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:51:25.092442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.092421    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:51:25.141342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.141293    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn" podStartSLOduration=1.141278966 podStartE2EDuration="1.141278966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:25.138043481 +0000 UTC m=+3279.391997106" watchObservedRunningTime="2026-11-02 15:51:25.141278966 +0000 UTC m=+3279.395232611"
Nov 02 15:51:25.715975 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:25.715937    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:26.110434 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:26.110349    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" event={"ID":"89d220ea-b8e7-492d-9036-68c13c08ece9","Type":"ContainerStarted","Data":"bc58b66f2c4bdd03f49a02f2cea518634de7aae57e2fde07ec137b2dfc8db4bf"}
Nov 02 15:51:26.112222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:26.112193    7019 generic.go:356] "Generic (PLEG): container finished" podID="71e48d42-f103-4187-8d90-4c4beed3c35e" containerID="d9376ce1f480cf34152700a51c359bc8543fd67a0cf5b66e916988409c745f12" exitCode=0
Nov 02 15:51:26.112340 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:26.112281    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn" event={"ID":"71e48d42-f103-4187-8d90-4c4beed3c35e","Type":"ContainerDied","Data":"d9376ce1f480cf34152700a51c359bc8543fd67a0cf5b66e916988409c745f12"}
Nov 02 15:51:26.152857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:26.152827    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:51:26.761709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:26.761676    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:51:26.763261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:26.763233    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:51:26.763384 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:26.763284    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 15:51:27.505891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.505860    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:27.544357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.544322    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:27.632721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.632690    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-openshift-ca\") pod \"71e48d42-f103-4187-8d90-4c4beed3c35e\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") "
Nov 02 15:51:27.633442 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.633422    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e48d42-f103-4187-8d90-4c4beed3c35e-kube-api-access-z5qnj\" (UniqueName: \"kubernetes.io/projected/71e48d42-f103-4187-8d90-4c4beed3c35e-kube-api-access-z5qnj\") pod \"71e48d42-f103-4187-8d90-4c4beed3c35e\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") "
Nov 02 15:51:27.633586 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.633573    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-xks-ca\") pod \"71e48d42-f103-4187-8d90-4c4beed3c35e\" (UID: \"71e48d42-f103-4187-8d90-4c4beed3c35e\") "
Nov 02 15:51:27.634193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.634151    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-openshift-ca" pod "71e48d42-f103-4187-8d90-4c4beed3c35e" (UID: "71e48d42-f103-4187-8d90-4c4beed3c35e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:27.636513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.636486    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e48d42-f103-4187-8d90-4c4beed3c35e-kube-api-access-z5qnj" pod "71e48d42-f103-4187-8d90-4c4beed3c35e" (UID: "71e48d42-f103-4187-8d90-4c4beed3c35e"). InnerVolumeSpecName "kube-api-access-z5qnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:27.636598 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.636565    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-xks-ca" pod "71e48d42-f103-4187-8d90-4c4beed3c35e" (UID: "71e48d42-f103-4187-8d90-4c4beed3c35e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:27.734924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.734837    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qnj\" (UniqueName: \"kubernetes.io/projected/71e48d42-f103-4187-8d90-4c4beed3c35e-kube-api-access-z5qnj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:27.734924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.734868    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:27.734924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.734880    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e48d42-f103-4187-8d90-4c4beed3c35e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:27.762098 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.762064    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:51:27.762497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:27.762129    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 15:51:28.127794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:28.127752    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn" event={"ID":"71e48d42-f103-4187-8d90-4c4beed3c35e","Type":"ContainerDied","Data":"268c2eb210a844e83744874bf9aef73a19322e90093c904b85f60730ea37964f"}
Nov 02 15:51:28.127981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:28.127819    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="268c2eb210a844e83744874bf9aef73a19322e90093c904b85f60730ea37964f"
Nov 02 15:51:28.127981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:28.127769    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:28.761867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:28.761828    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:51:28.762061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:28.761902    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 15:51:29.206268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:29.206229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:29.352118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:29.352068    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:29.502218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:29.502132    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:29.766216 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:29.766124    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:51:29.766499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:29.766469    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 15:51:30.749545 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:30.749509    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:30.762306 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:30.762275    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 02 15:51:30.762576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:30.762528    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 02 15:51:31.766589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:31.766558    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:51:31.767797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:31.767774    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:51:32.423917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:32.423876    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:34.515824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:34.515788    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:34.667801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:34.667622    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:34.820212 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:34.820102    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:36.173813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:36.173780    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:36.938868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:36.938828    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:37.080280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:37.080245    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:37.758494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:37.758461    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:39.819440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:39.819384    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:40.159625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:40.159519    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:40.596514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:40.596407    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:40.885173 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:40.885142    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:41.167723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:41.163437    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:41.630766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:41.630733    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:42.827359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:42.827310    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:43.304947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:43.304895    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:45.260973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:45.260921    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:45.743903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:45.743866    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:45.851158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:45.851122    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"]
Nov 02 15:51:45.864415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:45.864385    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"]
Nov 02 15:51:45.873019 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:45.872982    7019 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn\" is forbidden: User \"system:node:ip-10-0-129-45.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-45.ec2.internal' and this object" podUID="71e48d42-f103-4187-8d90-4c4beed3c35e" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-8a3fb0-h74gn"
Nov 02 15:51:45.906682 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:45.906627    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:46.058996 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:46.058913    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:46.331205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:46.331110    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:47.277470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.277427    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:47.474192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.474154    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:47.618680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.616465    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:47.756556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.754792    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:47.863968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.863925    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e48d42-f103-4187-8d90-4c4beed3c35e" path="/var/lib/kubelet/pods/71e48d42-f103-4187-8d90-4c4beed3c35e/volumes"
Nov 02 15:51:47.891278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.891192    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"]
Nov 02 15:51:47.902346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.902302    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:47.902535 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:47.902326    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d79991-2bdjp"]
Nov 02 15:51:48.246191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:48.246093    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:48.358717 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:48.358675    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"]
Nov 02 15:51:48.371061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:48.371015    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-1cc1b2-vw65n"]
Nov 02 15:51:48.390498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:48.390462    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:48.533526 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:48.533443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:48.852771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:48.852681    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:49.250604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:49.250567    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:49.388406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:49.388373    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:49.862876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:49.862842    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70eedf6e-3a0f-4c3a-8743-cde6b687d17f" path="/var/lib/kubelet/pods/70eedf6e-3a0f-4c3a-8743-cde6b687d17f/volumes"
Nov 02 15:51:49.863406 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:49.863387    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a74621-ad4e-4585-b658-10bfde829ba1" path="/var/lib/kubelet/pods/b0a74621-ad4e-4585-b658-10bfde829ba1/volumes"
Nov 02 15:51:50.674246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:50.674209    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:50.837276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:50.837241    7019 scope.go:122] "RemoveContainer" containerID="b4c8aa292aea7f44c1c3a8adcba341332cc13c26f225e82cfdd5d7b57087c2da"
Nov 02 15:51:50.848218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:50.848197    7019 scope.go:122] "RemoveContainer" containerID="b8a68f2c989bd77946cb7c3c2c5782afbf438a04506d4b0a00b235b4eac136f6"
Nov 02 15:51:50.859475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:50.859449    7019 scope.go:122] "RemoveContainer" containerID="89de9ca07b9711c135441dcf79ce5504c930ce7394bfe7a9269ea26a441899a7"
Nov 02 15:51:50.872693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:50.872662    7019 scope.go:122] "RemoveContainer" containerID="d7f2ecb220e5b0131de2dc4e5a48cf9c88b73dac319c985d8030d4d2bb4e39f6"
Nov 02 15:51:50.882625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:50.882602    7019 scope.go:122] "RemoveContainer" containerID="644864ccddfea93c90b2c509beb1a5c3629899441d64d53ec48bd9cac83bbebe"
Nov 02 15:51:52.174016 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:52.173975    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:52.708398 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:51:52.708361    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:51:52.871884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:52.871843    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:53.024838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.024742    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:53.207923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.207885    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:53.713767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.713723    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"]
Nov 02 15:51:53.714364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.714349    7019 cpu_manager.go:414] "containerMap: removing container" podUID="71e48d42-f103-4187-8d90-4c4beed3c35e" containerName="revoke-keys"
Nov 02 15:51:53.714426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.714366    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e48d42-f103-4187-8d90-4c4beed3c35e" containerName="revoke-keys"
Nov 02 15:51:53.714491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.714481    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e48d42-f103-4187-8d90-4c4beed3c35e" containerName="revoke-keys"
Nov 02 15:51:53.719843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.719816    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.738154 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.738126    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"]
Nov 02 15:51:53.762566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.762530    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:51:53.762755 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.762540    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:51:53.785362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.785311    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:53.827792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.827752    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.828175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.828149    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.828337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.828320    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hstj\" (UniqueName: \"kubernetes.io/projected/dcc83f75-c9cf-4b11-a654-9f642900741d-kube-api-access-2hstj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.929323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.929278    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.929517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.929374    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.929517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.929391    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hstj\" (UniqueName: \"kubernetes.io/projected/dcc83f75-c9cf-4b11-a654-9f642900741d-kube-api-access-2hstj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.929993 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.929964    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.931762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.931740    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:53.953806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:53.953775    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hstj\" (UniqueName: \"kubernetes.io/projected/dcc83f75-c9cf-4b11-a654-9f642900741d-kube-api-access-2hstj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:54.034869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:54.034779    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:54.348135 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:51:54.348102    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc83f75_c9cf_4b11_a654_9f642900741d.slice/crio-1fe82b98959eefd9fd4a213db9edf7b25a67f7868d6603cb758f82842d489201 WatchSource:0}: Error finding container 1fe82b98959eefd9fd4a213db9edf7b25a67f7868d6603cb758f82842d489201: Status 404 returned error can't find the container with id 1fe82b98959eefd9fd4a213db9edf7b25a67f7868d6603cb758f82842d489201
Nov 02 15:51:54.349786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:54.349757    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"]
Nov 02 15:51:54.441692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:54.441612    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4" event={"ID":"dcc83f75-c9cf-4b11-a654-9f642900741d","Type":"ContainerStarted","Data":"1fe82b98959eefd9fd4a213db9edf7b25a67f7868d6603cb758f82842d489201"}
Nov 02 15:51:55.450231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:55.450194    7019 generic.go:356] "Generic (PLEG): container finished" podID="dcc83f75-c9cf-4b11-a654-9f642900741d" containerID="d8ddea6226bad38c80dcbf20a810a283ba0659c1660c07b8e70da5a2086b6630" exitCode=0
Nov 02 15:51:55.450666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:55.450282    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4" event={"ID":"dcc83f75-c9cf-4b11-a654-9f642900741d","Type":"ContainerDied","Data":"d8ddea6226bad38c80dcbf20a810a283ba0659c1660c07b8e70da5a2086b6630"}
Nov 02 15:51:55.467143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:55.467107    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:55.758087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:55.758048    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:55.922071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:55.922032    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:56.092724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.085329    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:56.286236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.286199    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:56.437257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.437224    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:56.485427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.483444    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"]
Nov 02 15:51:56.488411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.488387    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.509312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.508486    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"]
Nov 02 15:51:56.578463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.578421    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.578694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.578493    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzf4n\" (UniqueName: \"kubernetes.io/projected/65e8eae7-ddac-429a-8934-7fd76bd79127-kube-api-access-tzf4n\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.578694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.578557    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.648789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.648764    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:56.680259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.680221    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.680479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.680318    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzf4n\" (UniqueName: \"kubernetes.io/projected/65e8eae7-ddac-429a-8934-7fd76bd79127-kube-api-access-tzf4n\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.680479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.680410    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.681262 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.681234    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.682937 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.682910    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.706638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.706556    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzf4n\" (UniqueName: \"kubernetes.io/projected/65e8eae7-ddac-429a-8934-7fd76bd79127-kube-api-access-tzf4n\") pod \"maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.781411 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.781374    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-xks-ca\") pod \"dcc83f75-c9cf-4b11-a654-9f642900741d\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") "
Nov 02 15:51:56.781681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.781627    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc83f75-c9cf-4b11-a654-9f642900741d-kube-api-access-2hstj\" (UniqueName: \"kubernetes.io/projected/dcc83f75-c9cf-4b11-a654-9f642900741d-kube-api-access-2hstj\") pod \"dcc83f75-c9cf-4b11-a654-9f642900741d\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") "
Nov 02 15:51:56.781797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.781778    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-openshift-ca\") pod \"dcc83f75-c9cf-4b11-a654-9f642900741d\" (UID: \"dcc83f75-c9cf-4b11-a654-9f642900741d\") "
Nov 02 15:51:56.782292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.782244    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-openshift-ca" pod "dcc83f75-c9cf-4b11-a654-9f642900741d" (UID: "dcc83f75-c9cf-4b11-a654-9f642900741d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:56.784087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.784060    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc83f75-c9cf-4b11-a654-9f642900741d-kube-api-access-2hstj" pod "dcc83f75-c9cf-4b11-a654-9f642900741d" (UID: "dcc83f75-c9cf-4b11-a654-9f642900741d"). InnerVolumeSpecName "kube-api-access-2hstj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:56.784247 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.784144    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-xks-ca" pod "dcc83f75-c9cf-4b11-a654-9f642900741d" (UID: "dcc83f75-c9cf-4b11-a654-9f642900741d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:56.833994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.833955    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:56.884828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.884786    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hstj\" (UniqueName: \"kubernetes.io/projected/dcc83f75-c9cf-4b11-a654-9f642900741d-kube-api-access-2hstj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:56.884828 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.884825    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:56.885064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.884841    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc83f75-c9cf-4b11-a654-9f642900741d-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:56.890010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:56.889979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:57.174444 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:51:57.174414    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e8eae7_ddac_429a_8934_7fd76bd79127.slice/crio-e9e9b27c8f38a8a3e5192f92622745d4152126e32f6b3cacc6e13dd430b6406c WatchSource:0}: Error finding container e9e9b27c8f38a8a3e5192f92622745d4152126e32f6b3cacc6e13dd430b6406c: Status 404 returned error can't find the container with id e9e9b27c8f38a8a3e5192f92622745d4152126e32f6b3cacc6e13dd430b6406c
Nov 02 15:51:57.176257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:57.176233    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"]
Nov 02 15:51:57.359689 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:57.359636    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:57.466288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:57.466193    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79" event={"ID":"65e8eae7-ddac-429a-8934-7fd76bd79127","Type":"ContainerStarted","Data":"e9e9b27c8f38a8a3e5192f92622745d4152126e32f6b3cacc6e13dd430b6406c"}
Nov 02 15:51:57.469692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:57.468165    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"
Nov 02 15:51:57.469692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:57.468216    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4" event={"ID":"dcc83f75-c9cf-4b11-a654-9f642900741d","Type":"ContainerDied","Data":"1fe82b98959eefd9fd4a213db9edf7b25a67f7868d6603cb758f82842d489201"}
Nov 02 15:51:57.469692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:57.468269    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fe82b98959eefd9fd4a213db9edf7b25a67f7868d6603cb758f82842d489201"
Nov 02 15:51:58.480611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:58.480575    7019 generic.go:356] "Generic (PLEG): container finished" podID="65e8eae7-ddac-429a-8934-7fd76bd79127" containerID="34ecbfd5c5203abd634ba8a3ff97f6276e51893c2a24741626c1dec972cdead3" exitCode=0
Nov 02 15:51:58.481180 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:58.480720    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79" event={"ID":"65e8eae7-ddac-429a-8934-7fd76bd79127","Type":"ContainerDied","Data":"34ecbfd5c5203abd634ba8a3ff97f6276e51893c2a24741626c1dec972cdead3"}
Nov 02 15:51:58.588751 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:58.588668    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:58.734344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:58.734311    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:59.598293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.598207    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:51:59.677325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.677294    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:51:59.849285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.849184    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-xks-ca\") pod \"65e8eae7-ddac-429a-8934-7fd76bd79127\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") "
Nov 02 15:51:59.849285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.849267    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-openshift-ca\") pod \"65e8eae7-ddac-429a-8934-7fd76bd79127\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") "
Nov 02 15:51:59.849527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.849306    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e8eae7-ddac-429a-8934-7fd76bd79127-kube-api-access-tzf4n\" (UniqueName: \"kubernetes.io/projected/65e8eae7-ddac-429a-8934-7fd76bd79127-kube-api-access-tzf4n\") pod \"65e8eae7-ddac-429a-8934-7fd76bd79127\" (UID: \"65e8eae7-ddac-429a-8934-7fd76bd79127\") "
Nov 02 15:51:59.849763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.849707    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-openshift-ca" pod "65e8eae7-ddac-429a-8934-7fd76bd79127" (UID: "65e8eae7-ddac-429a-8934-7fd76bd79127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:51:59.850181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.850157    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:59.852191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.852160    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-xks-ca" pod "65e8eae7-ddac-429a-8934-7fd76bd79127" (UID: "65e8eae7-ddac-429a-8934-7fd76bd79127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:51:59.860781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.860744    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e8eae7-ddac-429a-8934-7fd76bd79127-kube-api-access-tzf4n" pod "65e8eae7-ddac-429a-8934-7fd76bd79127" (UID: "65e8eae7-ddac-429a-8934-7fd76bd79127"). InnerVolumeSpecName "kube-api-access-tzf4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:51:59.951703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.951660    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e8eae7-ddac-429a-8934-7fd76bd79127-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:59.951703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.951700    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzf4n\" (UniqueName: \"kubernetes.io/projected/65e8eae7-ddac-429a-8934-7fd76bd79127-kube-api-access-tzf4n\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:51:59.964044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:51:59.964006    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:52:00.499344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:00.499300    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"
Nov 02 15:52:00.499344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:00.499323    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79" event={"ID":"65e8eae7-ddac-429a-8934-7fd76bd79127","Type":"ContainerDied","Data":"e9e9b27c8f38a8a3e5192f92622745d4152126e32f6b3cacc6e13dd430b6406c"}
Nov 02 15:52:00.499634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:00.499438    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e9b27c8f38a8a3e5192f92622745d4152126e32f6b3cacc6e13dd430b6406c"
Nov 02 15:52:01.240889 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:01.240844    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:02.846379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:02.846337    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:04.340392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:04.340355    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:04.491923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:04.491879    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:06.500299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:06.500264    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:07.046668 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:07.046622    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"]
Nov 02 15:52:07.057832 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:07.057801    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46794ced-wdvtd"]
Nov 02 15:52:07.862513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:07.862478    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c75944-01cb-423c-bfae-0046bee77d68" path="/var/lib/kubelet/pods/55c75944-01cb-423c-bfae-0046bee77d68/volumes"
Nov 02 15:52:07.984971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:07.984935    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:09.365048 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:09.365005    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:09.612422 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:09.612381    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:11.740838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:11.740799    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:13.330522 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:13.330480    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:14.462641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:14.462602    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:15.128142 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:15.128107    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:16.884897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.884850    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"]
Nov 02 15:52:16.886119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.886088    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dcc83f75-c9cf-4b11-a654-9f642900741d" containerName="revoke-keys"
Nov 02 15:52:16.886119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.886120    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc83f75-c9cf-4b11-a654-9f642900741d" containerName="revoke-keys"
Nov 02 15:52:16.886312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.886274    7019 cpu_manager.go:414] "containerMap: removing container" podUID="65e8eae7-ddac-429a-8934-7fd76bd79127" containerName="revoke-keys"
Nov 02 15:52:16.886312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.886283    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e8eae7-ddac-429a-8934-7fd76bd79127" containerName="revoke-keys"
Nov 02 15:52:16.886560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.886542    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e8eae7-ddac-429a-8934-7fd76bd79127" containerName="revoke-keys"
Nov 02 15:52:16.886622 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.886571    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc83f75-c9cf-4b11-a654-9f642900741d" containerName="revoke-keys"
Nov 02 15:52:16.891960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.891937    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:16.906361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.906331    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"]
Nov 02 15:52:16.915175 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.915144    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:52:16.915375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:16.915296    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:52:17.013531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.013498    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.013734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.013545    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkqq4\" (UniqueName: \"kubernetes.io/projected/bdd9afc2-db93-4c97-90ed-aecf99f72abc-kube-api-access-rkqq4\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.013820 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.013775    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.032331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.032298    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"]
Nov 02 15:52:17.035049 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.035027    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e54315a-4q7cj"]
Nov 02 15:52:17.115088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.115041    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.115302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.115146    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.115302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.115194    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkqq4\" (UniqueName: \"kubernetes.io/projected/bdd9afc2-db93-4c97-90ed-aecf99f72abc-kube-api-access-rkqq4\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.115903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.115884    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.117487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.117470    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.141450 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.141379    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkqq4\" (UniqueName: \"kubernetes.io/projected/bdd9afc2-db93-4c97-90ed-aecf99f72abc-kube-api-access-rkqq4\") pod \"maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.185972 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.185769    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:17.209323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.209294    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:17.520409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.520370    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"]
Nov 02 15:52:17.643137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.643100    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt" event={"ID":"bdd9afc2-db93-4c97-90ed-aecf99f72abc","Type":"ContainerStarted","Data":"20a6907b14db10aa05d814b702402991b097cd2a506d673e3a92f4f443264182"}
Nov 02 15:52:17.862383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:17.862347    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501cd36a-77b8-429b-b253-949f798f9a9d" path="/var/lib/kubelet/pods/501cd36a-77b8-429b-b253-949f798f9a9d/volumes"
Nov 02 15:52:18.651660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:18.651623    7019 generic.go:356] "Generic (PLEG): container finished" podID="bdd9afc2-db93-4c97-90ed-aecf99f72abc" containerID="255f06216255fc0af29ab3469f5b7230329dc13e28e81e63215dce9ce836d256" exitCode=0
Nov 02 15:52:18.652056 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:18.651711    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt" event={"ID":"bdd9afc2-db93-4c97-90ed-aecf99f72abc","Type":"ContainerDied","Data":"255f06216255fc0af29ab3469f5b7230329dc13e28e81e63215dce9ce836d256"}
Nov 02 15:52:19.005276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.005180    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:19.851339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.851305    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:19.953809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.953721    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-openshift-ca\") pod \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") "
Nov 02 15:52:19.953977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.953842    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd9afc2-db93-4c97-90ed-aecf99f72abc-kube-api-access-rkqq4\" (UniqueName: \"kubernetes.io/projected/bdd9afc2-db93-4c97-90ed-aecf99f72abc-kube-api-access-rkqq4\") pod \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") "
Nov 02 15:52:19.953977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.953898    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-xks-ca\") pod \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\" (UID: \"bdd9afc2-db93-4c97-90ed-aecf99f72abc\") "
Nov 02 15:52:19.954152 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.954125    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-openshift-ca" pod "bdd9afc2-db93-4c97-90ed-aecf99f72abc" (UID: "bdd9afc2-db93-4c97-90ed-aecf99f72abc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:19.954794 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.954769    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:19.956488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.956459    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-xks-ca" pod "bdd9afc2-db93-4c97-90ed-aecf99f72abc" (UID: "bdd9afc2-db93-4c97-90ed-aecf99f72abc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:19.956488 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:19.956459    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd9afc2-db93-4c97-90ed-aecf99f72abc-kube-api-access-rkqq4" pod "bdd9afc2-db93-4c97-90ed-aecf99f72abc" (UID: "bdd9afc2-db93-4c97-90ed-aecf99f72abc"). InnerVolumeSpecName "kube-api-access-rkqq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:20.056462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:20.056430    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkqq4\" (UniqueName: \"kubernetes.io/projected/bdd9afc2-db93-4c97-90ed-aecf99f72abc-kube-api-access-rkqq4\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:20.056462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:20.056458    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdd9afc2-db93-4c97-90ed-aecf99f72abc-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:20.667568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:20.667540    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"
Nov 02 15:52:20.667788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:20.667579    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt" event={"ID":"bdd9afc2-db93-4c97-90ed-aecf99f72abc","Type":"ContainerDied","Data":"20a6907b14db10aa05d814b702402991b097cd2a506d673e3a92f4f443264182"}
Nov 02 15:52:20.667788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:20.667624    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20a6907b14db10aa05d814b702402991b097cd2a506d673e3a92f4f443264182"
Nov 02 15:52:21.192798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.192076    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:21.862192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.862161    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"]
Nov 02 15:52:21.862730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.862706    7019 cpu_manager.go:414] "containerMap: removing container" podUID="bdd9afc2-db93-4c97-90ed-aecf99f72abc" containerName="revoke-keys"
Nov 02 15:52:21.862730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.862722    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd9afc2-db93-4c97-90ed-aecf99f72abc" containerName="revoke-keys"
Nov 02 15:52:21.862862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.862851    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd9afc2-db93-4c97-90ed-aecf99f72abc" containerName="revoke-keys"
Nov 02 15:52:21.866507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.866490    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:21.886578 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.886544    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"]
Nov 02 15:52:21.894375 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.894346    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:52:21.894772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.894748    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:52:21.983640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.983603    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:21.983823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.983672    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:21.983985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:21.983963    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rmd\" (UniqueName: \"kubernetes.io/projected/01865a2a-a627-45b5-a4ed-7369083799f8-kube-api-access-h8rmd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.085382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.085345    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rmd\" (UniqueName: \"kubernetes.io/projected/01865a2a-a627-45b5-a4ed-7369083799f8-kube-api-access-h8rmd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.085603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.085471    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.085603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.085503    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.086238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.086205    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.088184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.088158    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.110193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.110159    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rmd\" (UniqueName: \"kubernetes.io/projected/01865a2a-a627-45b5-a4ed-7369083799f8-kube-api-access-h8rmd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.181307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.181215    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:22.486145 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:52:22.486104    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01865a2a_a627_45b5_a4ed_7369083799f8.slice/crio-7b0306f673c25f6819a3c76d36f6ca6ff5d8cd630f36ea47d88c1908712b31c0 WatchSource:0}: Error finding container 7b0306f673c25f6819a3c76d36f6ca6ff5d8cd630f36ea47d88c1908712b31c0: Status 404 returned error can't find the container with id 7b0306f673c25f6819a3c76d36f6ca6ff5d8cd630f36ea47d88c1908712b31c0
Nov 02 15:52:22.488268 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.488235    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"]
Nov 02 15:52:22.679582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.679561    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:22.690131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:22.690092    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s" event={"ID":"01865a2a-a627-45b5-a4ed-7369083799f8","Type":"ContainerStarted","Data":"7b0306f673c25f6819a3c76d36f6ca6ff5d8cd630f36ea47d88c1908712b31c0"}
Nov 02 15:52:22.713447 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:22.713388    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:52:23.132246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.132213    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:23.284864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.284780    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:23.334344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.334308    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"]
Nov 02 15:52:23.344849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.344817    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-453864-cd6v4"]
Nov 02 15:52:23.439462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.439426    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:23.700300 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.700265    7019 generic.go:356] "Generic (PLEG): container finished" podID="01865a2a-a627-45b5-a4ed-7369083799f8" containerID="d80d845f5b51ce397d24268765180437843c235fe1e9374e780221baafe9493b" exitCode=0
Nov 02 15:52:23.700833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.700366    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s" event={"ID":"01865a2a-a627-45b5-a4ed-7369083799f8","Type":"ContainerDied","Data":"d80d845f5b51ce397d24268765180437843c235fe1e9374e780221baafe9493b"}
Nov 02 15:52:23.836018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.835986    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:23.862257 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:23.862227    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc83f75-c9cf-4b11-a654-9f642900741d" path="/var/lib/kubelet/pods/dcc83f75-c9cf-4b11-a654-9f642900741d/volumes"
Nov 02 15:52:24.872452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:24.872426    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:24.968028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:24.967935    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01865a2a-a627-45b5-a4ed-7369083799f8-kube-api-access-h8rmd\" (UniqueName: \"kubernetes.io/projected/01865a2a-a627-45b5-a4ed-7369083799f8-kube-api-access-h8rmd\") pod \"01865a2a-a627-45b5-a4ed-7369083799f8\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") "
Nov 02 15:52:24.968028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:24.967980    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-xks-ca\") pod \"01865a2a-a627-45b5-a4ed-7369083799f8\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") "
Nov 02 15:52:24.968271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:24.968074    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-openshift-ca\") pod \"01865a2a-a627-45b5-a4ed-7369083799f8\" (UID: \"01865a2a-a627-45b5-a4ed-7369083799f8\") "
Nov 02 15:52:24.968774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:24.968746    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-openshift-ca" pod "01865a2a-a627-45b5-a4ed-7369083799f8" (UID: "01865a2a-a627-45b5-a4ed-7369083799f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:24.970583 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:24.970551    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01865a2a-a627-45b5-a4ed-7369083799f8-kube-api-access-h8rmd" pod "01865a2a-a627-45b5-a4ed-7369083799f8" (UID: "01865a2a-a627-45b5-a4ed-7369083799f8"). InnerVolumeSpecName "kube-api-access-h8rmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:24.970762 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:24.970619    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-xks-ca" pod "01865a2a-a627-45b5-a4ed-7369083799f8" (UID: "01865a2a-a627-45b5-a4ed-7369083799f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:25.019308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.019260    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:25.070210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.070174    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:25.070210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.070210    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01865a2a-a627-45b5-a4ed-7369083799f8-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:25.070459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.070225    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rmd\" (UniqueName: \"kubernetes.io/projected/01865a2a-a627-45b5-a4ed-7369083799f8-kube-api-access-h8rmd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:25.394404 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.394361    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:52:25.763726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.756513    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s" event={"ID":"01865a2a-a627-45b5-a4ed-7369083799f8","Type":"ContainerDied","Data":"7b0306f673c25f6819a3c76d36f6ca6ff5d8cd630f36ea47d88c1908712b31c0"}
Nov 02 15:52:25.763726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.756579    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0306f673c25f6819a3c76d36f6ca6ff5d8cd630f36ea47d88c1908712b31c0"
Nov 02 15:52:25.763726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:25.756670    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"
Nov 02 15:52:26.657179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:26.657130    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:26.796571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:26.796538    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:28.844356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:28.844317    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:30.215393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:30.215355    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:30.427449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:30.427413    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:30.517334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:30.517232    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"]
Nov 02 15:52:30.517529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:30.517505    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy" containerID="cri-o://bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5" gracePeriod=30
Nov 02 15:52:30.638671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:30.621509    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:31.021640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.021606    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:31.231161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.231126    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"]
Nov 02 15:52:31.231732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.231716    7019 cpu_manager.go:414] "containerMap: removing container" podUID="01865a2a-a627-45b5-a4ed-7369083799f8" containerName="revoke-keys"
Nov 02 15:52:31.231798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.231734    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01865a2a-a627-45b5-a4ed-7369083799f8" containerName="revoke-keys"
Nov 02 15:52:31.231861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.231848    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="01865a2a-a627-45b5-a4ed-7369083799f8" containerName="revoke-keys"
Nov 02 15:52:31.236883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.236862    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.255979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.255946    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"]
Nov 02 15:52:31.268060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.268025    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqct\" (UniqueName: \"kubernetes.io/projected/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-kube-api-access-hmqct\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.268248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.268073    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.268248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.268092    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.277569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.277487    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:52:31.277569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.277535    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:52:31.369638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.369595    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqct\" (UniqueName: \"kubernetes.io/projected/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-kube-api-access-hmqct\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.369864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.369685    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.369864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.369713    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.370449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.370425    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.372239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.372214    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.394230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.394200    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqct\" (UniqueName: \"kubernetes.io/projected/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-kube-api-access-hmqct\") pod \"maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.551178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.551094    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:31.859510 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:52:31.859480    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94cb7f7_a299_43cb_94b8_6b9c60abf8a6.slice/crio-5b89404bbac4f7cb31a681bbab9982f16df0c8efdc133e7b9cd5323ff75aef7c WatchSource:0}: Error finding container 5b89404bbac4f7cb31a681bbab9982f16df0c8efdc133e7b9cd5323ff75aef7c: Status 404 returned error can't find the container with id 5b89404bbac4f7cb31a681bbab9982f16df0c8efdc133e7b9cd5323ff75aef7c
Nov 02 15:52:31.878195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.878161    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"]
Nov 02 15:52:31.959907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:31.959862    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:32.217674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:32.216228    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:32.840950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:32.840907    7019 generic.go:356] "Generic (PLEG): container finished" podID="b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" containerID="31b35ceb6846f1f52d1c10a0e32c24fd8332a9fac700d5e522c66eff5f6f6c83" exitCode=0
Nov 02 15:52:32.841353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:32.840993    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4" event={"ID":"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6","Type":"ContainerDied","Data":"31b35ceb6846f1f52d1c10a0e32c24fd8332a9fac700d5e522c66eff5f6f6c83"}
Nov 02 15:52:32.841353 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:32.841053    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4" event={"ID":"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6","Type":"ContainerStarted","Data":"5b89404bbac4f7cb31a681bbab9982f16df0c8efdc133e7b9cd5323ff75aef7c"}
Nov 02 15:52:33.562537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:33.562499    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:33.743623 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:33.743584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:33.885190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:33.885156    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:34.060292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.060253    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"]
Nov 02 15:52:34.131547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.093820    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0f849b1-vkq2h"]
Nov 02 15:52:34.152259 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.152235    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:34.160316 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.160085    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:34.233994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.233964    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-xks-ca\") pod \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") "
Nov 02 15:52:34.234215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.234200    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-kube-api-access-hmqct\" (UniqueName: \"kubernetes.io/projected/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-kube-api-access-hmqct\") pod \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") "
Nov 02 15:52:34.234292 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.234236    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-openshift-ca\") pod \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\" (UID: \"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6\") "
Nov 02 15:52:34.234766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.234732    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-openshift-ca" pod "b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" (UID: "b94cb7f7-a299-43cb-94b8-6b9c60abf8a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:34.236217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.236185    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-xks-ca" pod "b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" (UID: "b94cb7f7-a299-43cb-94b8-6b9c60abf8a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:34.236946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.236924    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-kube-api-access-hmqct" pod "b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" (UID: "b94cb7f7-a299-43cb-94b8-6b9c60abf8a6"). InnerVolumeSpecName "kube-api-access-hmqct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:34.258270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.258236    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"]
Nov 02 15:52:34.262155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.262122    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-af33a6-hff79"]
Nov 02 15:52:34.295669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.295133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:34.336974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.336233    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqct\" (UniqueName: \"kubernetes.io/projected/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-kube-api-access-hmqct\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.336974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.336309    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.336974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.336329    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:34.826096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.826058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:34.858638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.858609    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"
Nov 02 15:52:34.858638 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.858621    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4" event={"ID":"b94cb7f7-a299-43cb-94b8-6b9c60abf8a6","Type":"ContainerDied","Data":"5b89404bbac4f7cb31a681bbab9982f16df0c8efdc133e7b9cd5323ff75aef7c"}
Nov 02 15:52:34.858854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:34.858700    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b89404bbac4f7cb31a681bbab9982f16df0c8efdc133e7b9cd5323ff75aef7c"
Nov 02 15:52:35.032797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.032759    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"]
Nov 02 15:52:35.035513 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.035491    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96ff4281-c7jmw"]
Nov 02 15:52:35.578878 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:35.578842    7019 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf51c40_d05d_4778_ac8e_5b9de73d775c.slice/crio-bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf51c40_d05d_4778_ac8e_5b9de73d775c.slice/crio-conmon-bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 15:52:35.783161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.783133    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:52:35.864392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.864354    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istiod-ca-cert\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.864590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.864410    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-kube-api-access-b85cg\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-kube-api-access-b85cg\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.864710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.864668    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-data\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.864782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.864764    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istiod-ca-cert" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:35.864854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.864765    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-socket\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.864915 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.864850    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-token\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-token\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.865071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.865050    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-data" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.865508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.865485    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-socket" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.865924 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.865896    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-certs" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.869149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.869115    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-token" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:35.870744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.869609    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-kube-api-access-b85cg" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "kube-api-access-b85cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:35.870848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.870742    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-certs\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.870848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.870823    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-podinfo\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.870976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.870909    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-credential-socket\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.870976 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.870965    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-envoy\") pod \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\" (UID: \"ccf51c40-d05d-4778-ac8e-5b9de73d775c\") "
Nov 02 15:52:35.874060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.871691    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.874060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.871710    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.874060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.871725    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b85cg\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-kube-api-access-b85cg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.874060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.871750    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.874060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.871764    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.874060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.871777    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.874060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.872725    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-credential-socket" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.876165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.876134    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-envoy" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:35.878666 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.878625    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-podinfo" pod "ccf51c40-d05d-4778-ac8e-5b9de73d775c" (UID: "ccf51c40-d05d-4778-ac8e-5b9de73d775c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:35.905532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.905494    7019 generic.go:356] "Generic (PLEG): container finished" podID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerID="bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5" exitCode=0
Nov 02 15:52:35.905721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.905705    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"
Nov 02 15:52:35.906508 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.906479    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e8eae7-ddac-429a-8934-7fd76bd79127" path="/var/lib/kubelet/pods/65e8eae7-ddac-429a-8934-7fd76bd79127/volumes"
Nov 02 15:52:35.907402 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.907375    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a30d0a-1fb9-4093-a0f5-bdbc50a357cb" path="/var/lib/kubelet/pods/75a30d0a-1fb9-4093-a0f5-bdbc50a357cb/volumes"
Nov 02 15:52:35.910730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.910707    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e238251a-24ef-4390-8e3f-11ad8f0404f3" path="/var/lib/kubelet/pods/e238251a-24ef-4390-8e3f-11ad8f0404f3/volumes"
Nov 02 15:52:35.911518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.911491    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" event={"ID":"ccf51c40-d05d-4778-ac8e-5b9de73d775c","Type":"ContainerDied","Data":"bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5"}
Nov 02 15:52:35.911603 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.911551    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k" event={"ID":"ccf51c40-d05d-4778-ac8e-5b9de73d775c","Type":"ContainerDied","Data":"a4c3727fae8b3a9d77768a4141cdf188d35a1ea7974a71d0063817a55233b476"}
Nov 02 15:52:35.911714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.911675    7019 scope.go:122] "RemoveContainer" containerID="bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5"
Nov 02 15:52:35.928725 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.928705    7019 scope.go:122] "RemoveContainer" containerID="bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5"
Nov 02 15:52:35.929006 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:35.928982    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5\": container with ID starting with bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5 not found: ID does not exist" containerID="bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5"
Nov 02 15:52:35.929070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.929013    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5"} err="failed to get container status \"bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5\": rpc error: code = NotFound desc = could not find container \"bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5\": container with ID starting with bd20fa2f9cb11cd9932d859c4976638b75c8ab9382f8f5df51fa48d348bc61e5 not found: ID does not exist"
Nov 02 15:52:35.971145 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.971116    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"]
Nov 02 15:52:35.973364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.973341    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.973471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.973366    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.973471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.973381    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccf51c40-d05d-4778-ac8e-5b9de73d775c-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:35.976878 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.976851    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"]
Nov 02 15:52:35.979568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.979545    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-453864-openshift-default-77d4cd74c7-4v58k"]
Nov 02 15:52:35.997864 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:35.997841    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"]
Nov 02 15:52:36.016178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:36.016150    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"]
Nov 02 15:52:36.045183 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:36.045154    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:36.209732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:36.209607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:37.431083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.431046    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"]
Nov 02 15:52:37.431678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.431660    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy"
Nov 02 15:52:37.431732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.431679    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy"
Nov 02 15:52:37.431771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.431733    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" containerName="revoke-keys"
Nov 02 15:52:37.431771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.431738    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" containerName="revoke-keys"
Nov 02 15:52:37.431856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.431846    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" containerName="revoke-keys"
Nov 02 15:52:37.431897 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.431858    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" containerName="istio-proxy"
Nov 02 15:52:37.436847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.436828    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.454771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.454745    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"]
Nov 02 15:52:37.464148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.464126    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:52:37.464420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.464407    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:52:37.496001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.495950    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.496228 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.496192    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.496314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.496236    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85bql\" (UniqueName: \"kubernetes.io/projected/8981d766-11b7-48e8-8341-26589b76aab1-kube-api-access-85bql\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.597537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.597487    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85bql\" (UniqueName: \"kubernetes.io/projected/8981d766-11b7-48e8-8341-26589b76aab1-kube-api-access-85bql\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.597764 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.597701    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.597886 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.597871    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.598703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.598669    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.600238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.600209    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.622036 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.622009    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85bql\" (UniqueName: \"kubernetes.io/projected/8981d766-11b7-48e8-8341-26589b76aab1-kube-api-access-85bql\") pod \"maas-api-revoke-sub-9ad377c3-pc64k\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.751364 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.751272    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:37.796213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.796177    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:37.871031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:37.870988    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf51c40-d05d-4778-ac8e-5b9de73d775c" path="/var/lib/kubelet/pods/ccf51c40-d05d-4778-ac8e-5b9de73d775c/volumes"
Nov 02 15:52:38.096035 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:52:38.096003    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8981d766_11b7_48e8_8341_26589b76aab1.slice/crio-f7b591f32a1446509818124117909c0974dd817cef1b642a822d560bab538d91 WatchSource:0}: Error finding container f7b591f32a1446509818124117909c0974dd817cef1b642a822d560bab538d91: Status 404 returned error can't find the container with id f7b591f32a1446509818124117909c0974dd817cef1b642a822d560bab538d91
Nov 02 15:52:38.096896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:38.096866    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"]
Nov 02 15:52:38.935572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:38.935531    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k" event={"ID":"8981d766-11b7-48e8-8341-26589b76aab1","Type":"ContainerStarted","Data":"5dfc24c0ab440357aa032fa7cf253daca660e81c5335b5f10d1e5d485ddf3a1d"}
Nov 02 15:52:38.936018 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:38.935598    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k" event={"ID":"8981d766-11b7-48e8-8341-26589b76aab1","Type":"ContainerStarted","Data":"f7b591f32a1446509818124117909c0974dd817cef1b642a822d560bab538d91"}
Nov 02 15:52:38.979553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:38.979490    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k" podStartSLOduration=1.979443051 podStartE2EDuration="1.979443051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:38.977964894 +0000 UTC m=+3353.231918519" watchObservedRunningTime="2026-11-02 15:52:38.979443051 +0000 UTC m=+3353.233396676"
Nov 02 15:52:39.490492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:39.490451    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:39.945338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:39.945304    7019 generic.go:356] "Generic (PLEG): container finished" podID="8981d766-11b7-48e8-8341-26589b76aab1" containerID="5dfc24c0ab440357aa032fa7cf253daca660e81c5335b5f10d1e5d485ddf3a1d" exitCode=0
Nov 02 15:52:39.945785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:39.945386    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k" event={"ID":"8981d766-11b7-48e8-8341-26589b76aab1","Type":"ContainerDied","Data":"5dfc24c0ab440357aa032fa7cf253daca660e81c5335b5f10d1e5d485ddf3a1d"}
Nov 02 15:52:40.988031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:40.988001    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:41.172511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.172483    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:41.193398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.193363    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:41.281774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.281693    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8981d766-11b7-48e8-8341-26589b76aab1-kube-api-access-85bql\" (UniqueName: \"kubernetes.io/projected/8981d766-11b7-48e8-8341-26589b76aab1-kube-api-access-85bql\") pod \"8981d766-11b7-48e8-8341-26589b76aab1\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") "
Nov 02 15:52:41.281981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.281855    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-openshift-ca\") pod \"8981d766-11b7-48e8-8341-26589b76aab1\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") "
Nov 02 15:52:41.282063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.281977    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-xks-ca\") pod \"8981d766-11b7-48e8-8341-26589b76aab1\" (UID: \"8981d766-11b7-48e8-8341-26589b76aab1\") "
Nov 02 15:52:41.282609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.282202    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-openshift-ca" pod "8981d766-11b7-48e8-8341-26589b76aab1" (UID: "8981d766-11b7-48e8-8341-26589b76aab1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:41.282609 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.282577    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.283962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.283939    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8981d766-11b7-48e8-8341-26589b76aab1-kube-api-access-85bql" pod "8981d766-11b7-48e8-8341-26589b76aab1" (UID: "8981d766-11b7-48e8-8341-26589b76aab1"). InnerVolumeSpecName "kube-api-access-85bql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:41.284072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.284047    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-xks-ca" pod "8981d766-11b7-48e8-8341-26589b76aab1" (UID: "8981d766-11b7-48e8-8341-26589b76aab1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:41.339432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.339390    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:41.404732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.393719    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85bql\" (UniqueName: \"kubernetes.io/projected/8981d766-11b7-48e8-8341-26589b76aab1-kube-api-access-85bql\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.404732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.393759    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8981d766-11b7-48e8-8341-26589b76aab1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:41.663148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.663111    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:41.743729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.743692    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"]
Nov 02 15:52:41.755824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.755791    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e40940-hhlbt"]
Nov 02 15:52:41.830738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.830693    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:41.862684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.862630    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd9afc2-db93-4c97-90ed-aecf99f72abc" path="/var/lib/kubelet/pods/bdd9afc2-db93-4c97-90ed-aecf99f72abc/volumes"
Nov 02 15:52:41.961977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.961902    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"
Nov 02 15:52:41.961977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.961922    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k" event={"ID":"8981d766-11b7-48e8-8341-26589b76aab1","Type":"ContainerDied","Data":"f7b591f32a1446509818124117909c0974dd817cef1b642a822d560bab538d91"}
Nov 02 15:52:41.961977 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.961971    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b591f32a1446509818124117909c0974dd817cef1b642a822d560bab538d91"
Nov 02 15:52:41.992354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:41.992316    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:42.194309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:42.194268    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:42.347118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:42.347016    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:42.695713 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:42.695680    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:42.866238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:42.866202    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:43.022704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.022581    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"]
Nov 02 15:52:43.023194 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.022915    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy" containerID="cri-o://d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2" gracePeriod=30
Nov 02 15:52:43.054788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.054759    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:43.657276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.657241    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"]
Nov 02 15:52:43.657827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.657813    7019 cpu_manager.go:414] "containerMap: removing container" podUID="8981d766-11b7-48e8-8341-26589b76aab1" containerName="revoke-keys"
Nov 02 15:52:43.657880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.657828    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8981d766-11b7-48e8-8341-26589b76aab1" containerName="revoke-keys"
Nov 02 15:52:43.657954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.657945    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="8981d766-11b7-48e8-8341-26589b76aab1" containerName="revoke-keys"
Nov 02 15:52:43.661558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.661540    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.679846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.679817    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:52:43.679846 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.679831    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:52:43.683072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.683045    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"]
Nov 02 15:52:43.745737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.745701    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:43.839703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.839668    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.839938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.839916    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4thd4\" (UniqueName: \"kubernetes.io/projected/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-kube-api-access-4thd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.839990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.839952    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.941258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.941160    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4thd4\" (UniqueName: \"kubernetes.io/projected/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-kube-api-access-4thd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.941258 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.941213    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.941523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.941275    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.941919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.941895    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.943728 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.943708    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.965505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.965476    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4thd4\" (UniqueName: \"kubernetes.io/projected/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-kube-api-access-4thd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:43.977335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:43.977315    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:44.286921 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:52:44.286889    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc21c90f_d6c9_43f5_a90c_3695d13e0a33.slice/crio-0f5a401787cc5adbdebcecea0302f8c599b453512e32e08255d3f844ff61f96c WatchSource:0}: Error finding container 0f5a401787cc5adbdebcecea0302f8c599b453512e32e08255d3f844ff61f96c: Status 404 returned error can't find the container with id 0f5a401787cc5adbdebcecea0302f8c599b453512e32e08255d3f844ff61f96c
Nov 02 15:52:44.287624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:44.287519    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"]
Nov 02 15:52:45.001966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.001933    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp" event={"ID":"dc21c90f-d6c9-43f5-a90c-3695d13e0a33","Type":"ContainerStarted","Data":"97b811f9f8c0902f1cfee71e3b94c2d57c6e0d4a271845cac681a5d278c41258"}
Nov 02 15:52:45.002176 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.001984    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp" event={"ID":"dc21c90f-d6c9-43f5-a90c-3695d13e0a33","Type":"ContainerStarted","Data":"0f5a401787cc5adbdebcecea0302f8c599b453512e32e08255d3f844ff61f96c"}
Nov 02 15:52:45.045906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.045853    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp" podStartSLOduration=2.045839576 podStartE2EDuration="2.045839576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:45.045544528 +0000 UTC m=+3359.299498153" watchObservedRunningTime="2026-11-02 15:52:45.045839576 +0000 UTC m=+3359.299793199"
Nov 02 15:52:45.798515 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.798477    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"]
Nov 02 15:52:45.803756 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.803731    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.816096 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.816072    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"]
Nov 02 15:52:45.820502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.820254    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.820822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.820794    7019 patch_prober.go:29] interesting pod/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:52:45.820944 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.820851    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:52:45.829206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.829180    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"]
Nov 02 15:52:45.841624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.841593    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"]
Nov 02 15:52:45.863307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.863274    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwb8p\" (UniqueName: \"kubernetes.io/projected/f665ad06-d721-4d30-ba4d-7ad32c9638f1-kube-api-access-vwb8p\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.863494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.863469    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.863582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.863508    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.863582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.863550    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.863718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.863665    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcl8k\" (UniqueName: \"kubernetes.io/projected/c0b52288-5405-47fc-9f32-7fa34431d14e-kube-api-access-jcl8k\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.863718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.863705    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.964700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.964627    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.964905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.964710    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.964905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.964747    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.964905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.964797    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcl8k\" (UniqueName: \"kubernetes.io/projected/c0b52288-5405-47fc-9f32-7fa34431d14e-kube-api-access-jcl8k\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.964905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.964826    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.965139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.964992    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwb8p\" (UniqueName: \"kubernetes.io/projected/f665ad06-d721-4d30-ba4d-7ad32c9638f1-kube-api-access-vwb8p\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.965391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.965360    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.965776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.965751    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.967750 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.967710    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.967851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.967793    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:45.991283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.991242    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwb8p\" (UniqueName: \"kubernetes.io/projected/f665ad06-d721-4d30-ba4d-7ad32c9638f1-kube-api-access-vwb8p\") pod \"maas-api-revoke-sub-98212ddf-bbbzc\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:45.992190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:45.992164    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcl8k\" (UniqueName: \"kubernetes.io/projected/c0b52288-5405-47fc-9f32-7fa34431d14e-kube-api-access-jcl8k\") pod \"maas-api-revoke-sub-fbc7a634-rdxgd\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:46.013219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:46.013187    7019 generic.go:356] "Generic (PLEG): container finished" podID="dc21c90f-d6c9-43f5-a90c-3695d13e0a33" containerID="97b811f9f8c0902f1cfee71e3b94c2d57c6e0d4a271845cac681a5d278c41258" exitCode=0
Nov 02 15:52:46.013377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:46.013274    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp" event={"ID":"dc21c90f-d6c9-43f5-a90c-3695d13e0a33","Type":"ContainerDied","Data":"97b811f9f8c0902f1cfee71e3b94c2d57c6e0d4a271845cac681a5d278c41258"}
Nov 02 15:52:46.119277 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:46.119247    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:46.141551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:46.141522    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:46.250677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:46.246400    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:46.433990 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:52:46.433961    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b52288_5405_47fc_9f32_7fa34431d14e.slice/crio-9a416b63016c0cc009b21258ea82cacee436de307fd0a3d67059125d8160e8a4 WatchSource:0}: Error finding container 9a416b63016c0cc009b21258ea82cacee436de307fd0a3d67059125d8160e8a4: Status 404 returned error can't find the container with id 9a416b63016c0cc009b21258ea82cacee436de307fd0a3d67059125d8160e8a4
Nov 02 15:52:46.435479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:46.435454    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"]
Nov 02 15:52:46.469702 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:52:46.469670    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf665ad06_d721_4d30_ba4d_7ad32c9638f1.slice/crio-daf3cf8a573112c6828d3c1d562c9f316587c12f50f3ff9c1f485da45eca47c8 WatchSource:0}: Error finding container daf3cf8a573112c6828d3c1d562c9f316587c12f50f3ff9c1f485da45eca47c8: Status 404 returned error can't find the container with id daf3cf8a573112c6828d3c1d562c9f316587c12f50f3ff9c1f485da45eca47c8
Nov 02 15:52:46.471594 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:46.471569    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"]
Nov 02 15:52:47.022926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.022886    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd" event={"ID":"c0b52288-5405-47fc-9f32-7fa34431d14e","Type":"ContainerStarted","Data":"fd246a0fed08a09b8b596c50c3ccc29f52e5be959f9f4ece4d5c8e9fd9d96de0"}
Nov 02 15:52:47.023383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.022951    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd" event={"ID":"c0b52288-5405-47fc-9f32-7fa34431d14e","Type":"ContainerStarted","Data":"9a416b63016c0cc009b21258ea82cacee436de307fd0a3d67059125d8160e8a4"}
Nov 02 15:52:47.024678 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.024639    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc" event={"ID":"f665ad06-d721-4d30-ba4d-7ad32c9638f1","Type":"ContainerStarted","Data":"7fbab9c76f871b07f869cf33006385f0e3640400549c3351a38f3e2d6cdbd2ea"}
Nov 02 15:52:47.024783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.024701    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc" event={"ID":"f665ad06-d721-4d30-ba4d-7ad32c9638f1","Type":"ContainerStarted","Data":"daf3cf8a573112c6828d3c1d562c9f316587c12f50f3ff9c1f485da45eca47c8"}
Nov 02 15:52:47.123179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.123114    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc" podStartSLOduration=2.123094991 podStartE2EDuration="2.123094991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:47.120338033 +0000 UTC m=+3361.374291658" watchObservedRunningTime="2026-11-02 15:52:47.123094991 +0000 UTC m=+3361.377048616"
Nov 02 15:52:47.123974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.123918    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd" podStartSLOduration=2.123902249 podStartE2EDuration="2.123902249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:47.070384352 +0000 UTC m=+3361.324337976" watchObservedRunningTime="2026-11-02 15:52:47.123902249 +0000 UTC m=+3361.377855873"
Nov 02 15:52:47.195054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.195026    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:47.283445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.283404    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-xks-ca\") pod \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") "
Nov 02 15:52:47.283445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.283449    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-openshift-ca\") pod \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") "
Nov 02 15:52:47.283732 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.283473    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-kube-api-access-4thd4\" (UniqueName: \"kubernetes.io/projected/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-kube-api-access-4thd4\") pod \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\" (UID: \"dc21c90f-d6c9-43f5-a90c-3695d13e0a33\") "
Nov 02 15:52:47.283861 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.283836    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-openshift-ca" pod "dc21c90f-d6c9-43f5-a90c-3695d13e0a33" (UID: "dc21c90f-d6c9-43f5-a90c-3695d13e0a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:47.284071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.284052    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:47.285700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.285664    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-kube-api-access-4thd4" pod "dc21c90f-d6c9-43f5-a90c-3695d13e0a33" (UID: "dc21c90f-d6c9-43f5-a90c-3695d13e0a33"). InnerVolumeSpecName "kube-api-access-4thd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:47.285831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.285749    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-xks-ca" pod "dc21c90f-d6c9-43f5-a90c-3695d13e0a33" (UID: "dc21c90f-d6c9-43f5-a90c-3695d13e0a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:47.385276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.385238    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:47.385276 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.385276    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4thd4\" (UniqueName: \"kubernetes.io/projected/dc21c90f-d6c9-43f5-a90c-3695d13e0a33-kube-api-access-4thd4\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:47.872323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:47.872141    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:48.035558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.035403    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"
Nov 02 15:52:48.035558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.035430    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp" event={"ID":"dc21c90f-d6c9-43f5-a90c-3695d13e0a33","Type":"ContainerDied","Data":"0f5a401787cc5adbdebcecea0302f8c599b453512e32e08255d3f844ff61f96c"}
Nov 02 15:52:48.035558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.035483    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5a401787cc5adbdebcecea0302f8c599b453512e32e08255d3f844ff61f96c"
Nov 02 15:52:48.037271 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.037246    7019 generic.go:356] "Generic (PLEG): container finished" podID="f665ad06-d721-4d30-ba4d-7ad32c9638f1" containerID="7fbab9c76f871b07f869cf33006385f0e3640400549c3351a38f3e2d6cdbd2ea" exitCode=0
Nov 02 15:52:48.037432 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.037406    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc" event={"ID":"f665ad06-d721-4d30-ba4d-7ad32c9638f1","Type":"ContainerDied","Data":"7fbab9c76f871b07f869cf33006385f0e3640400549c3351a38f3e2d6cdbd2ea"}
Nov 02 15:52:48.039368 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.039344    7019 generic.go:356] "Generic (PLEG): container finished" podID="c0b52288-5405-47fc-9f32-7fa34431d14e" containerID="fd246a0fed08a09b8b596c50c3ccc29f52e5be959f9f4ece4d5c8e9fd9d96de0" exitCode=0
Nov 02 15:52:48.039469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.039376    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd" event={"ID":"c0b52288-5405-47fc-9f32-7fa34431d14e","Type":"ContainerDied","Data":"fd246a0fed08a09b8b596c50c3ccc29f52e5be959f9f4ece4d5c8e9fd9d96de0"}
Nov 02 15:52:48.324171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.324143    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:52:48.406104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406018    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-credential-socket\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406290 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406234    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-podinfo\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406306    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istiod-ca-cert\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406359 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406328    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-credential-socket" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.406465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406398    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-envoy\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406439    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-data\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406468    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-socket\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406521    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-kube-api-access-7fnnq\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-kube-api-access-7fnnq\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406558    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-token\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-token\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406624    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-certs\") pod \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\" (UID: \"47b3fdc1-62fc-438e-aebe-c77c3b960f10\") "
Nov 02 15:52:48.406865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.406813    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istiod-ca-cert" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:48.407144 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.407069    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-certs" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.407230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.407168    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-socket" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.407297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.407228    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-data" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.407297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.407274    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.407531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.407306    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.407531 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.407475    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.409126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.409080    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-podinfo" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:48.409738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.409700    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-kube-api-access-7fnnq" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "kube-api-access-7fnnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:48.409928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.409912    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-envoy" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:48.410239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.410220    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-token" pod "47b3fdc1-62fc-438e-aebe-c77c3b960f10" (UID: "47b3fdc1-62fc-438e-aebe-c77c3b960f10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:48.508325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.508287    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.508325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.508318    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.508325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.508327    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.508325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.508335    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47b3fdc1-62fc-438e-aebe-c77c3b960f10-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.508597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.508344    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fnnq\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-kube-api-access-7fnnq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.508597 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.508353    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47b3fdc1-62fc-438e-aebe-c77c3b960f10-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:48.829885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.829788    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:48.988604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:48.988565    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:49.048787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.048749    7019 generic.go:356] "Generic (PLEG): container finished" podID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerID="d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2" exitCode=0
Nov 02 15:52:49.049289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.048816    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"
Nov 02 15:52:49.049289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.048828    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" event={"ID":"47b3fdc1-62fc-438e-aebe-c77c3b960f10","Type":"ContainerDied","Data":"d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2"}
Nov 02 15:52:49.049289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.048884    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47" event={"ID":"47b3fdc1-62fc-438e-aebe-c77c3b960f10","Type":"ContainerDied","Data":"a84d16a7d842c8c3c2ada785870be5b8d11400903a9bac1e6e0088a2fbeae88b"}
Nov 02 15:52:49.049289 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.048909    7019 scope.go:122] "RemoveContainer" containerID="d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2"
Nov 02 15:52:49.061539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.061516    7019 scope.go:122] "RemoveContainer" containerID="f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b"
Nov 02 15:52:49.077128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.077023    7019 scope.go:122] "RemoveContainer" containerID="d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2"
Nov 02 15:52:49.077432 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:49.077411    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2\": container with ID starting with d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2 not found: ID does not exist" containerID="d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2"
Nov 02 15:52:49.077537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.077446    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2"} err="failed to get container status \"d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2\": rpc error: code = NotFound desc = could not find container \"d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2\": container with ID starting with d91eb72b1166b611f3d81862828ff3d5daeef62e119fc146987c447a612b1bc2 not found: ID does not exist"
Nov 02 15:52:49.077537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.077472    7019 scope.go:122] "RemoveContainer" containerID="f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b"
Nov 02 15:52:49.077919 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:49.077898    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b\": container with ID starting with f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b not found: ID does not exist" containerID="f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b"
Nov 02 15:52:49.078021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.077953    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b"} err="failed to get container status \"f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b\": rpc error: code = NotFound desc = could not find container \"f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b\": container with ID starting with f5a074a178b1f13a3940e6ee8a05526ddd150b82c93286c1eafa2bde94b2967b not found: ID does not exist"
Nov 02 15:52:49.128936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.128899    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"]
Nov 02 15:52:49.134136 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.134084    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-af33a6-openshift-default-759865cbb9-26m47"]
Nov 02 15:52:49.237838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.237809    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:49.268875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.268849    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:49.319278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.319238    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-openshift-ca\") pod \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") "
Nov 02 15:52:49.319565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.319343    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-xks-ca\") pod \"c0b52288-5405-47fc-9f32-7fa34431d14e\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") "
Nov 02 15:52:49.319565 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.319462    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-xks-ca\") pod \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") "
Nov 02 15:52:49.319848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.319627    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-openshift-ca\") pod \"c0b52288-5405-47fc-9f32-7fa34431d14e\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") "
Nov 02 15:52:49.319848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.319677    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f665ad06-d721-4d30-ba4d-7ad32c9638f1-kube-api-access-vwb8p\" (UniqueName: \"kubernetes.io/projected/f665ad06-d721-4d30-ba4d-7ad32c9638f1-kube-api-access-vwb8p\") pod \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\" (UID: \"f665ad06-d721-4d30-ba4d-7ad32c9638f1\") "
Nov 02 15:52:49.319848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.319705    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b52288-5405-47fc-9f32-7fa34431d14e-kube-api-access-jcl8k\" (UniqueName: \"kubernetes.io/projected/c0b52288-5405-47fc-9f32-7fa34431d14e-kube-api-access-jcl8k\") pod \"c0b52288-5405-47fc-9f32-7fa34431d14e\" (UID: \"c0b52288-5405-47fc-9f32-7fa34431d14e\") "
Nov 02 15:52:49.320134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.320103    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-openshift-ca" pod "c0b52288-5405-47fc-9f32-7fa34431d14e" (UID: "c0b52288-5405-47fc-9f32-7fa34431d14e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:49.320315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.320292    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:49.321073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.321050    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-openshift-ca" pod "f665ad06-d721-4d30-ba4d-7ad32c9638f1" (UID: "f665ad06-d721-4d30-ba4d-7ad32c9638f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:49.326129 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.326104    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b52288-5405-47fc-9f32-7fa34431d14e-kube-api-access-jcl8k" pod "c0b52288-5405-47fc-9f32-7fa34431d14e" (UID: "c0b52288-5405-47fc-9f32-7fa34431d14e"). InnerVolumeSpecName "kube-api-access-jcl8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:49.327054 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.327024    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-xks-ca" pod "c0b52288-5405-47fc-9f32-7fa34431d14e" (UID: "c0b52288-5405-47fc-9f32-7fa34431d14e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:49.328440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.328418    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-xks-ca" pod "f665ad06-d721-4d30-ba4d-7ad32c9638f1" (UID: "f665ad06-d721-4d30-ba4d-7ad32c9638f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:49.328816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.328786    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f665ad06-d721-4d30-ba4d-7ad32c9638f1-kube-api-access-vwb8p" pod "f665ad06-d721-4d30-ba4d-7ad32c9638f1" (UID: "f665ad06-d721-4d30-ba4d-7ad32c9638f1"). InnerVolumeSpecName "kube-api-access-vwb8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:49.421854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.421770    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwb8p\" (UniqueName: \"kubernetes.io/projected/f665ad06-d721-4d30-ba4d-7ad32c9638f1-kube-api-access-vwb8p\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:49.421854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.421802    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcl8k\" (UniqueName: \"kubernetes.io/projected/c0b52288-5405-47fc-9f32-7fa34431d14e-kube-api-access-jcl8k\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:49.421854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.421812    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:49.421854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.421822    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b52288-5405-47fc-9f32-7fa34431d14e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:49.421854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.421831    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f665ad06-d721-4d30-ba4d-7ad32c9638f1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:49.500553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.500509    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:49.539628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.539577    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"]
Nov 02 15:52:49.540312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.540097    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy" containerID="cri-o://2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81" gracePeriod=30
Nov 02 15:52:49.680758 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.673229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:49.863395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:49.863359    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" path="/var/lib/kubelet/pods/47b3fdc1-62fc-438e-aebe-c77c3b960f10/volumes"
Nov 02 15:52:50.057125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.057098    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"
Nov 02 15:52:50.057590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.057134    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd" event={"ID":"c0b52288-5405-47fc-9f32-7fa34431d14e","Type":"ContainerDied","Data":"9a416b63016c0cc009b21258ea82cacee436de307fd0a3d67059125d8160e8a4"}
Nov 02 15:52:50.057590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.057188    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a416b63016c0cc009b21258ea82cacee436de307fd0a3d67059125d8160e8a4"
Nov 02 15:52:50.058879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.058860    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"
Nov 02 15:52:50.058879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.058879    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc" event={"ID":"f665ad06-d721-4d30-ba4d-7ad32c9638f1","Type":"ContainerDied","Data":"daf3cf8a573112c6828d3c1d562c9f316587c12f50f3ff9c1f485da45eca47c8"}
Nov 02 15:52:50.059064 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.058919    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daf3cf8a573112c6828d3c1d562c9f316587c12f50f3ff9c1f485da45eca47c8"
Nov 02 15:52:50.445266 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445225    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"]
Nov 02 15:52:50.445839 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445824    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c0b52288-5405-47fc-9f32-7fa34431d14e" containerName="revoke-keys"
Nov 02 15:52:50.445899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445840    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b52288-5405-47fc-9f32-7fa34431d14e" containerName="revoke-keys"
Nov 02 15:52:50.445899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445852    7019 cpu_manager.go:414] "containerMap: removing container" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy"
Nov 02 15:52:50.445899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445857    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy"
Nov 02 15:52:50.445899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445867    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f665ad06-d721-4d30-ba4d-7ad32c9638f1" containerName="revoke-keys"
Nov 02 15:52:50.445899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445872    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f665ad06-d721-4d30-ba4d-7ad32c9638f1" containerName="revoke-keys"
Nov 02 15:52:50.445899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445881    7019 cpu_manager.go:414] "containerMap: removing container" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy"
Nov 02 15:52:50.445899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445885    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy"
Nov 02 15:52:50.446117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445912    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dc21c90f-d6c9-43f5-a90c-3695d13e0a33" containerName="revoke-keys"
Nov 02 15:52:50.446117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.445917    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc21c90f-d6c9-43f5-a90c-3695d13e0a33" containerName="revoke-keys"
Nov 02 15:52:50.446117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.446006    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc21c90f-d6c9-43f5-a90c-3695d13e0a33" containerName="revoke-keys"
Nov 02 15:52:50.446117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.446017    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b52288-5405-47fc-9f32-7fa34431d14e" containerName="revoke-keys"
Nov 02 15:52:50.446117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.446025    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy"
Nov 02 15:52:50.446117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.446031    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f665ad06-d721-4d30-ba4d-7ad32c9638f1" containerName="revoke-keys"
Nov 02 15:52:50.446117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.446038    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b3fdc1-62fc-438e-aebe-c77c3b960f10" containerName="istio-proxy"
Nov 02 15:52:50.450613 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.450592    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.467504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.466933    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:52:50.467504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.467139    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:52:50.471069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.471042    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"]
Nov 02 15:52:50.540441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.540396    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.540726 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.540705    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.540798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.540754    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lnvg\" (UniqueName: \"kubernetes.io/projected/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-kube-api-access-5lnvg\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.641925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.641881    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.642125 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.642101    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.642199 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.642149    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lnvg\" (UniqueName: \"kubernetes.io/projected/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-kube-api-access-5lnvg\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.642602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.642576    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.644615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.644588    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.668178 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.668149    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lnvg\" (UniqueName: \"kubernetes.io/projected/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-kube-api-access-5lnvg\") pod \"maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.768813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.768727    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:50.951528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.951492    7019 scope.go:122] "RemoveContainer" containerID="f3f0d43afa26611d53fcfd424804754daf6f0c36be20d369e5bd7e2e15a7ea36"
Nov 02 15:52:50.962272 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:50.962248    7019 scope.go:122] "RemoveContainer" containerID="f3660d9e5009f185cafc4acd850d995d21e40d9df13b058f82ddf33661913959"
Nov 02 15:52:51.000892 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.000857    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" containerID="cri-o://d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f" gracePeriod=45
Nov 02 15:52:51.010255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.010227    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused"
Nov 02 15:52:51.019514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.019458    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main" containerID="cri-o://3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7" gracePeriod=45
Nov 02 15:52:51.039160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.039130    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main" containerID="cri-o://a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00" gracePeriod=45
Nov 02 15:52:51.074797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.074765    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"]
Nov 02 15:52:51.075137 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:52:51.075014    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc300c5c6_12ea_41ae_b677_14a3cf4cc0e3.slice/crio-2d416c1020c3351f82cf389ff0588e9bef1447ebd3cf9248981aff88b686073d WatchSource:0}: Error finding container 2d416c1020c3351f82cf389ff0588e9bef1447ebd3cf9248981aff88b686073d: Status 404 returned error can't find the container with id 2d416c1020c3351f82cf389ff0588e9bef1447ebd3cf9248981aff88b686073d
Nov 02 15:52:51.402201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.400558    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:52:51.403425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.403396    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:52:51.407489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.407471    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453368    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-model-cache\") pod \"832acbfc-6909-41ec-9aed-b653a7d9a662\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453424    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-dshm\") pod \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453475    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-dshm\") pod \"f9a6d0fb-4834-47b4-928f-7baf205291fc\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453514    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9a6d0fb-4834-47b4-928f-7baf205291fc-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a6d0fb-4834-47b4-928f-7baf205291fc-tls-certs\") pod \"f9a6d0fb-4834-47b4-928f-7baf205291fc\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453550    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tmp-dir\") pod \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453580    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-home\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-home\") pod \"f9a6d0fb-4834-47b4-928f-7baf205291fc\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453706    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832acbfc-6909-41ec-9aed-b653a7d9a662-kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/832acbfc-6909-41ec-9aed-b653a7d9a662-kube-api-access-m9mvt\") pod \"832acbfc-6909-41ec-9aed-b653a7d9a662\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453790    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9a6d0fb-4834-47b4-928f-7baf205291fc-kube-api-access-sr4g8\" (UniqueName: \"kubernetes.io/projected/f9a6d0fb-4834-47b4-928f-7baf205291fc-kube-api-access-sr4g8\") pod \"f9a6d0fb-4834-47b4-928f-7baf205291fc\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453834    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-model-cache\") pod \"f9a6d0fb-4834-47b4-928f-7baf205291fc\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453873    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-tmp-dir\") pod \"f9a6d0fb-4834-47b4-928f-7baf205291fc\" (UID: \"f9a6d0fb-4834-47b4-928f-7baf205291fc\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453950    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-tmp-dir\") pod \"832acbfc-6909-41ec-9aed-b653a7d9a662\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.453998    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tls-certs\") pod \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.454089    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-home\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-home\") pod \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.454124    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-model-cache\") pod \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.454176    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60cb7c62-9ce5-4a85-96df-00a6ea763e49-kube-api-access-stlhp\" (UniqueName: \"kubernetes.io/projected/60cb7c62-9ce5-4a85-96df-00a6ea763e49-kube-api-access-stlhp\") pod \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\" (UID: \"60cb7c62-9ce5-4a85-96df-00a6ea763e49\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.454224    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-dshm\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-dshm\") pod \"832acbfc-6909-41ec-9aed-b653a7d9a662\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.454254    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-home\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-home\") pod \"832acbfc-6909-41ec-9aed-b653a7d9a662\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") "
Nov 02 15:52:51.455314 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.454306    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/832acbfc-6909-41ec-9aed-b653a7d9a662-tls-certs\" (UniqueName: \"kubernetes.io/secret/832acbfc-6909-41ec-9aed-b653a7d9a662-tls-certs\") pod \"832acbfc-6909-41ec-9aed-b653a7d9a662\" (UID: \"832acbfc-6909-41ec-9aed-b653a7d9a662\") "
Nov 02 15:52:51.456630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.455730    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-model-cache" pod "f9a6d0fb-4834-47b4-928f-7baf205291fc" (UID: "f9a6d0fb-4834-47b4-928f-7baf205291fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.456630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.456173    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-tmp-dir" pod "f9a6d0fb-4834-47b4-928f-7baf205291fc" (UID: "f9a6d0fb-4834-47b4-928f-7baf205291fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.457382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.457173    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-model-cache" pod "60cb7c62-9ce5-4a85-96df-00a6ea763e49" (UID: "60cb7c62-9ce5-4a85-96df-00a6ea763e49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.457683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.457537    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-home" pod "832acbfc-6909-41ec-9aed-b653a7d9a662" (UID: "832acbfc-6909-41ec-9aed-b653a7d9a662"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.457769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.457729    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tmp-dir" pod "60cb7c62-9ce5-4a85-96df-00a6ea763e49" (UID: "60cb7c62-9ce5-4a85-96df-00a6ea763e49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.458320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.458027    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-home" pod "f9a6d0fb-4834-47b4-928f-7baf205291fc" (UID: "f9a6d0fb-4834-47b4-928f-7baf205291fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.458851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.458830    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-tmp-dir" pod "832acbfc-6909-41ec-9aed-b653a7d9a662" (UID: "832acbfc-6909-41ec-9aed-b653a7d9a662"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.461829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.455735    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-home" pod "60cb7c62-9ce5-4a85-96df-00a6ea763e49" (UID: "60cb7c62-9ce5-4a85-96df-00a6ea763e49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.462107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.462084    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-model-cache" pod "832acbfc-6909-41ec-9aed-b653a7d9a662" (UID: "832acbfc-6909-41ec-9aed-b653a7d9a662"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.464474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.464445    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-dshm" pod "832acbfc-6909-41ec-9aed-b653a7d9a662" (UID: "832acbfc-6909-41ec-9aed-b653a7d9a662"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.464981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.464952    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-dshm" pod "f9a6d0fb-4834-47b4-928f-7baf205291fc" (UID: "f9a6d0fb-4834-47b4-928f-7baf205291fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.465217 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.465196    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-dshm" pod "60cb7c62-9ce5-4a85-96df-00a6ea763e49" (UID: "60cb7c62-9ce5-4a85-96df-00a6ea763e49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:51.466076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.465970    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832acbfc-6909-41ec-9aed-b653a7d9a662-tls-certs" pod "832acbfc-6909-41ec-9aed-b653a7d9a662" (UID: "832acbfc-6909-41ec-9aed-b653a7d9a662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:51.466076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.466041    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832acbfc-6909-41ec-9aed-b653a7d9a662-kube-api-access-m9mvt" pod "832acbfc-6909-41ec-9aed-b653a7d9a662" (UID: "832acbfc-6909-41ec-9aed-b653a7d9a662"). InnerVolumeSpecName "kube-api-access-m9mvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:51.466076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.466065    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tls-certs" pod "60cb7c62-9ce5-4a85-96df-00a6ea763e49" (UID: "60cb7c62-9ce5-4a85-96df-00a6ea763e49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:51.466408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.466388    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9a6d0fb-4834-47b4-928f-7baf205291fc-tls-certs" pod "f9a6d0fb-4834-47b4-928f-7baf205291fc" (UID: "f9a6d0fb-4834-47b4-928f-7baf205291fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:51.467506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.467479    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cb7c62-9ce5-4a85-96df-00a6ea763e49-kube-api-access-stlhp" pod "60cb7c62-9ce5-4a85-96df-00a6ea763e49" (UID: "60cb7c62-9ce5-4a85-96df-00a6ea763e49"). InnerVolumeSpecName "kube-api-access-stlhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:51.467612 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.467545    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a6d0fb-4834-47b4-928f-7baf205291fc-kube-api-access-sr4g8" pod "f9a6d0fb-4834-47b4-928f-7baf205291fc" (UID: "f9a6d0fb-4834-47b4-928f-7baf205291fc"). InnerVolumeSpecName "kube-api-access-sr4g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:51.556837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556795    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.556837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556828    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.556837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556837    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/832acbfc-6909-41ec-9aed-b653a7d9a662-kube-api-access-m9mvt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556850    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr4g8\" (UniqueName: \"kubernetes.io/projected/f9a6d0fb-4834-47b4-928f-7baf205291fc-kube-api-access-sr4g8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556860    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556868    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556875    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556884    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60cb7c62-9ce5-4a85-96df-00a6ea763e49-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556891    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556900    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556909    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stlhp\" (UniqueName: \"kubernetes.io/projected/60cb7c62-9ce5-4a85-96df-00a6ea763e49-kube-api-access-stlhp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556917    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556925    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556932    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832acbfc-6909-41ec-9aed-b653a7d9a662-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556940    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/832acbfc-6909-41ec-9aed-b653a7d9a662-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556948    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60cb7c62-9ce5-4a85-96df-00a6ea763e49-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556955    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9a6d0fb-4834-47b4-928f-7baf205291fc-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.557170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.556962    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9a6d0fb-4834-47b4-928f-7baf205291fc-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:51.688311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:51.688205    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:52.077829 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.077784    7019 generic.go:356] "Generic (PLEG): container finished" podID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerID="d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f" exitCode=0
Nov 02 15:52:52.078329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.077841    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"
Nov 02 15:52:52.078329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.077865    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" event={"ID":"f9a6d0fb-4834-47b4-928f-7baf205291fc","Type":"ContainerDied","Data":"d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f"}
Nov 02 15:52:52.078329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.077905    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg" event={"ID":"f9a6d0fb-4834-47b4-928f-7baf205291fc","Type":"ContainerDied","Data":"db357c7b854d85ba281b0ffbb34088194d46a46be4d6f37e26da741fa772f22e"}
Nov 02 15:52:52.078329 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.077922    7019 scope.go:122] "RemoveContainer" containerID="d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f"
Nov 02 15:52:52.079543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.079522    7019 generic.go:356] "Generic (PLEG): container finished" podID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerID="3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7" exitCode=0
Nov 02 15:52:52.079752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.079634    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" event={"ID":"60cb7c62-9ce5-4a85-96df-00a6ea763e49","Type":"ContainerDied","Data":"3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7"}
Nov 02 15:52:52.079752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.079711    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"
Nov 02 15:52:52.079881 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.079714    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5" event={"ID":"60cb7c62-9ce5-4a85-96df-00a6ea763e49","Type":"ContainerDied","Data":"eb1f3ff3571920c3a65061b0eb98e6b77dcfee39591d6cd2c7a12e343543ca35"}
Nov 02 15:52:52.081437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.081412    7019 generic.go:356] "Generic (PLEG): container finished" podID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerID="a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00" exitCode=0
Nov 02 15:52:52.081581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.081464    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"
Nov 02 15:52:52.081581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.081501    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" event={"ID":"832acbfc-6909-41ec-9aed-b653a7d9a662","Type":"ContainerDied","Data":"a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00"}
Nov 02 15:52:52.081581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.081529    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p" event={"ID":"832acbfc-6909-41ec-9aed-b653a7d9a662","Type":"ContainerDied","Data":"a7a20c633993d95a07a0ed39a547e95d65c12e6ebc6ad918fd5df059eb50e0f9"}
Nov 02 15:52:52.083942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.083865    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2" event={"ID":"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3","Type":"ContainerStarted","Data":"32a0ad1ca01ce000db95e452a90f59b1bdd20b5d50a4a0a260d638c3b43a8c6b"}
Nov 02 15:52:52.083942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.083938    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2" event={"ID":"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3","Type":"ContainerStarted","Data":"2d416c1020c3351f82cf389ff0588e9bef1447ebd3cf9248981aff88b686073d"}
Nov 02 15:52:52.092299 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.092279    7019 scope.go:122] "RemoveContainer" containerID="d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f"
Nov 02 15:52:52.092569 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:52.092547    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f\": container with ID starting with d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f not found: ID does not exist" containerID="d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f"
Nov 02 15:52:52.092660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.092580    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f"} err="failed to get container status \"d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f\": rpc error: code = NotFound desc = could not find container \"d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f\": container with ID starting with d0b5781218b1ebe81194f89e5fbb80f2df0ec2388194d90ca50c0db5c8d85f5f not found: ID does not exist"
Nov 02 15:52:52.092660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.092603    7019 scope.go:122] "RemoveContainer" containerID="3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7"
Nov 02 15:52:52.101806 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.101788    7019 scope.go:122] "RemoveContainer" containerID="3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7"
Nov 02 15:52:52.102067 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:52.102048    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7\": container with ID starting with 3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7 not found: ID does not exist" containerID="3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7"
Nov 02 15:52:52.102131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.102075    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7"} err="failed to get container status \"3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7\": rpc error: code = NotFound desc = could not find container \"3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7\": container with ID starting with 3505b0664e706332be31c3bfa5aeff46c6de7bed85bf9c8aca080ff751f289b7 not found: ID does not exist"
Nov 02 15:52:52.102131 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.102091    7019 scope.go:122] "RemoveContainer" containerID="a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00"
Nov 02 15:52:52.112179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.112159    7019 scope.go:122] "RemoveContainer" containerID="a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00"
Nov 02 15:52:52.112454 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:52.112436    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00\": container with ID starting with a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00 not found: ID does not exist" containerID="a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00"
Nov 02 15:52:52.112518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.112460    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00"} err="failed to get container status \"a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00\": rpc error: code = NotFound desc = could not find container \"a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00\": container with ID starting with a373a0f6e6a4634a588b33d2c9dea9dba784457d0495ce171afdfb90bcce0b00 not found: ID does not exist"
Nov 02 15:52:52.134787 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.134761    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"]
Nov 02 15:52:52.147467 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.147441    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/distinct-2-w0-6af50f-kserve-dcf8c8ffb-mxvwg"]
Nov 02 15:52:52.183163 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.183136    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"]
Nov 02 15:52:52.194414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.194384    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/simulator-w0-6af50f-kserve-78f797bdc7-qmf2p"]
Nov 02 15:52:52.228865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.228832    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"]
Nov 02 15:52:52.233291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.233262    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6af50f/premium-w0-6af50f-kserve-758c89b79b-zcdt5"]
Nov 02 15:52:52.274330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:52.274219    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2" podStartSLOduration=2.2742057239999998 podStartE2EDuration="2.274205724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:52:52.271759992 +0000 UTC m=+3366.525713615" watchObservedRunningTime="2026-11-02 15:52:52.274205724 +0000 UTC m=+3366.528159371"
Nov 02 15:52:52.710749 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:52.710709    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:52:53.095464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:53.095371    7019 generic.go:356] "Generic (PLEG): container finished" podID="c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" containerID="32a0ad1ca01ce000db95e452a90f59b1bdd20b5d50a4a0a260d638c3b43a8c6b" exitCode=0
Nov 02 15:52:53.095884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:53.095452    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2" event={"ID":"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3","Type":"ContainerDied","Data":"32a0ad1ca01ce000db95e452a90f59b1bdd20b5d50a4a0a260d638c3b43a8c6b"}
Nov 02 15:52:53.319617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:53.319583    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:53.862914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:53.862881    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" path="/var/lib/kubelet/pods/60cb7c62-9ce5-4a85-96df-00a6ea763e49/volumes"
Nov 02 15:52:53.863395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:53.863379    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" path="/var/lib/kubelet/pods/832acbfc-6909-41ec-9aed-b653a7d9a662/volumes"
Nov 02 15:52:53.863805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:53.863792    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" path="/var/lib/kubelet/pods/f9a6d0fb-4834-47b4-928f-7baf205291fc/volumes"
Nov 02 15:52:54.287810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.287778    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:54.297224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.297198    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:54.400518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.400488    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-kube-api-access-5lnvg\" (UniqueName: \"kubernetes.io/projected/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-kube-api-access-5lnvg\") pod \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") "
Nov 02 15:52:54.400715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.400636    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-openshift-ca\") pod \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") "
Nov 02 15:52:54.401122 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.400808    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-xks-ca\") pod \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\" (UID: \"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3\") "
Nov 02 15:52:54.401706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.401642    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-openshift-ca" pod "c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" (UID: "c300c5c6-12ea-41ae-b677-14a3cf4cc0e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:54.403390 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.403362    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-kube-api-access-5lnvg" pod "c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" (UID: "c300c5c6-12ea-41ae-b677-14a3cf4cc0e3"). InnerVolumeSpecName "kube-api-access-5lnvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:54.403625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.403588    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-xks-ca" pod "c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" (UID: "c300c5c6-12ea-41ae-b677-14a3cf4cc0e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:52:54.431088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.431061    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:54.502919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.502887    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.502919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.502913    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lnvg\" (UniqueName: \"kubernetes.io/projected/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-kube-api-access-5lnvg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.502919 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.502924    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:54.902094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:54.902056    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:52:55.037989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.037900    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-envoy\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.037989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.037952    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-kube-api-access-fnsqp\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-kube-api-access-fnsqp\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.037991    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-data\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038052    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-socket\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038116    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istiod-ca-cert\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038153    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-credential-socket\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038195    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-certs\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038260    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-podinfo\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038304    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-token\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-token\") pod \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\" (UID: \"e394f8f8-2b26-4dcd-8fa9-84a62560e269\") "
Nov 02 15:52:55.038641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038610    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-credential-socket" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:55.038788 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.038763    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-data" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:55.039053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039025    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istiod-ca-cert" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:52:55.039156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039055    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-socket" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:55.039305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039277    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-certs" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:55.039857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039832    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.039857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039858    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.040025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039867    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.040025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039877    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.040025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.039892    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.041130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.041100    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-kube-api-access-fnsqp" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "kube-api-access-fnsqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:55.041448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.041397    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:55.041537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.041500    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-podinfo" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:52:55.041587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.041571    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-token" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:52:55.041945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.041923    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-envoy" pod "e394f8f8-2b26-4dcd-8fa9-84a62560e269" (UID: "e394f8f8-2b26-4dcd-8fa9-84a62560e269"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:52:55.112746 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.112709    7019 generic.go:356] "Generic (PLEG): container finished" podID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerID="2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81" exitCode=0
Nov 02 15:52:55.112947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.112760    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"
Nov 02 15:52:55.112947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.112765    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" event={"ID":"e394f8f8-2b26-4dcd-8fa9-84a62560e269","Type":"ContainerDied","Data":"2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81"}
Nov 02 15:52:55.112947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.112819    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c" event={"ID":"e394f8f8-2b26-4dcd-8fa9-84a62560e269","Type":"ContainerDied","Data":"29a1a132d7b2d7f32042d3b08279258df2dc1f41d753d343d9032689631d98e9"}
Nov 02 15:52:55.112947 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.112848    7019 scope.go:122] "RemoveContainer" containerID="2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81"
Nov 02 15:52:55.114414 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.114396    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"
Nov 02 15:52:55.114540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.114451    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2" event={"ID":"c300c5c6-12ea-41ae-b677-14a3cf4cc0e3","Type":"ContainerDied","Data":"2d416c1020c3351f82cf389ff0588e9bef1447ebd3cf9248981aff88b686073d"}
Nov 02 15:52:55.114540 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.114496    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d416c1020c3351f82cf389ff0588e9bef1447ebd3cf9248981aff88b686073d"
Nov 02 15:52:55.124291 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.124204    7019 scope.go:122] "RemoveContainer" containerID="2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81"
Nov 02 15:52:55.124610 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:52:55.124578    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81\": container with ID starting with 2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81 not found: ID does not exist" containerID="2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81"
Nov 02 15:52:55.124718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.124614    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81"} err="failed to get container status \"2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81\": rpc error: code = NotFound desc = could not find container \"2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81\": container with ID starting with 2bd9fdce460fd3f1de40cf343947fd9090759be8a0a1bd258f454c8541498c81 not found: ID does not exist"
Nov 02 15:52:55.141171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.141146    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.141171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.141171    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnsqp\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-kube-api-access-fnsqp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.141350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.141181    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.141350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.141190    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e394f8f8-2b26-4dcd-8fa9-84a62560e269-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:52:55.188576 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.188529    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"]
Nov 02 15:52:55.191130 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.191103    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e40940-openshift-default-6fb44b465f-9r52c"]
Nov 02 15:52:55.861445 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:55.861407    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" path="/var/lib/kubelet/pods/e394f8f8-2b26-4dcd-8fa9-84a62560e269/volumes"
Nov 02 15:52:56.036133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:56.036094    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:56.723872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:56.723239    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:52:58.418680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:52:58.418597    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:00.023538 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:00.022231    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:00.897203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:00.897165    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:53:01.273819 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.273718    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"]
Nov 02 15:53:01.274365 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.273982    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58" podUID="db681668-d7cc-4da3-8835-1b72ef6687f1" containerName="maas-api" containerID="cri-o://232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29" gracePeriod=30
Nov 02 15:53:01.390573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.390535    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:01.736684 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.733859    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:53:01.862358 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.862329    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-maas-api-tls\") pod \"db681668-d7cc-4da3-8835-1b72ef6687f1\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") "
Nov 02 15:53:01.862564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.862400    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db681668-d7cc-4da3-8835-1b72ef6687f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db681668-d7cc-4da3-8835-1b72ef6687f1-openshift-service-ca\") pod \"db681668-d7cc-4da3-8835-1b72ef6687f1\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") "
Nov 02 15:53:01.862564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.862432    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-metrics-certs\") pod \"db681668-d7cc-4da3-8835-1b72ef6687f1\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") "
Nov 02 15:53:01.862564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.862480    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db681668-d7cc-4da3-8835-1b72ef6687f1-kube-api-access-gj7z7\" (UniqueName: \"kubernetes.io/projected/db681668-d7cc-4da3-8835-1b72ef6687f1-kube-api-access-gj7z7\") pod \"db681668-d7cc-4da3-8835-1b72ef6687f1\" (UID: \"db681668-d7cc-4da3-8835-1b72ef6687f1\") "
Nov 02 15:53:01.862827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.862804    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db681668-d7cc-4da3-8835-1b72ef6687f1-openshift-service-ca" pod "db681668-d7cc-4da3-8835-1b72ef6687f1" (UID: "db681668-d7cc-4da3-8835-1b72ef6687f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:01.862929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.862912    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db681668-d7cc-4da3-8835-1b72ef6687f1-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:01.864585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.864562    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-metrics-certs" pod "db681668-d7cc-4da3-8835-1b72ef6687f1" (UID: "db681668-d7cc-4da3-8835-1b72ef6687f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:01.864772 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.864749    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-maas-api-tls" pod "db681668-d7cc-4da3-8835-1b72ef6687f1" (UID: "db681668-d7cc-4da3-8835-1b72ef6687f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:01.864824 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.864764    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db681668-d7cc-4da3-8835-1b72ef6687f1-kube-api-access-gj7z7" pod "db681668-d7cc-4da3-8835-1b72ef6687f1" (UID: "db681668-d7cc-4da3-8835-1b72ef6687f1"). InnerVolumeSpecName "kube-api-access-gj7z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:01.870059 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.869953    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:01.981032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.965202    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:01.981032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.965246    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj7z7\" (UniqueName: \"kubernetes.io/projected/db681668-d7cc-4da3-8835-1b72ef6687f1-kube-api-access-gj7z7\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:01.981032 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.965264    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db681668-d7cc-4da3-8835-1b72ef6687f1-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:01.992321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.992277    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"]
Nov 02 15:53:01.995814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:01.995790    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6f2e0c-57pcp"]
Nov 02 15:53:02.020768 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.020736    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:02.167963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.167926    7019 generic.go:356] "Generic (PLEG): container finished" podID="db681668-d7cc-4da3-8835-1b72ef6687f1" containerID="232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29" exitCode=0
Nov 02 15:53:02.168149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.167977    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"
Nov 02 15:53:02.168149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.168011    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58" event={"ID":"db681668-d7cc-4da3-8835-1b72ef6687f1","Type":"ContainerDied","Data":"232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29"}
Nov 02 15:53:02.168149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.168068    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58" event={"ID":"db681668-d7cc-4da3-8835-1b72ef6687f1","Type":"ContainerDied","Data":"1fa5f2ea48d80c5d093bea207825fd3294360c5a2e771e1408370e012cb226ae"}
Nov 02 15:53:02.168149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.168090    7019 scope.go:122] "RemoveContainer" containerID="232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29"
Nov 02 15:53:02.179709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.179689    7019 scope.go:122] "RemoveContainer" containerID="232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29"
Nov 02 15:53:02.179994 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:02.179976    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29\": container with ID starting with 232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29 not found: ID does not exist" containerID="232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29"
Nov 02 15:53:02.180070 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.180002    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29"} err="failed to get container status \"232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29\": rpc error: code = NotFound desc = could not find container \"232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29\": container with ID starting with 232e31398145a48b2e6a93f0de4de15d168ab072d4cc954cad81aa4eb1fafe29 not found: ID does not exist"
Nov 02 15:53:02.231378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.231279    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"]
Nov 02 15:53:02.242255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.242226    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6f2e0c-c9c4f5c8-4gb58"]
Nov 02 15:53:02.497084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.497003    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"]
Nov 02 15:53:02.497593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.497224    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy" containerID="cri-o://50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473" gracePeriod=30
Nov 02 15:53:02.603979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:02.603942    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:03.704060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:03.704024    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:03.862641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:03.862597    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db681668-d7cc-4da3-8835-1b72ef6687f1" path="/var/lib/kubelet/pods/db681668-d7cc-4da3-8835-1b72ef6687f1/volumes"
Nov 02 15:53:03.863102 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:03.863081    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc21c90f-d6c9-43f5-a90c-3695d13e0a33" path="/var/lib/kubelet/pods/dc21c90f-d6c9-43f5-a90c-3695d13e0a33/volumes"
Nov 02 15:53:04.011051 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:04.010964    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:04.294195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:04.294109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:04.479027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:04.478985    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:04.485850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:04.485816    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"]
Nov 02 15:53:04.497248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:04.497217    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0039c-nwt6s"]
Nov 02 15:53:04.652477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:04.652435    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:04.983749 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:04.983662    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:05.863428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:05.863396    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01865a2a-a627-45b5-a4ed-7369083799f8" path="/var/lib/kubelet/pods/01865a2a-a627-45b5-a4ed-7369083799f8/volumes"
Nov 02 15:53:06.457055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:06.457017    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:06.983917 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:06.983880    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:07.124496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.106284    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"]
Nov 02 15:53:07.124496 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.124452    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6af50f-s6bq4"]
Nov 02 15:53:07.158504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.158453    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:07.625862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.625826    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:07.774958 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.774876    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:53:07.863511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.863472    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94cb7f7-a299-43cb-94b8-6b9c60abf8a6" path="/var/lib/kubelet/pods/b94cb7f7-a299-43cb-94b8-6b9c60abf8a6/volumes"
Nov 02 15:53:07.918570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918526    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-podinfo\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.918786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918608    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-credential-socket\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.918786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918641    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-data\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.918941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918841    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-envoy\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.918941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918883    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589f2b68-0b46-4320-880d-ff9e9d047d18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589f2b68-0b46-4320-880d-ff9e9d047d18-istiod-ca-cert\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.918941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918938    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-token\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-token\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.919113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918964    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-certs\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.919113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.918987    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-credential-socket" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:07.919113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.919041    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-socket\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.919113 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.919096    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-kube-api-access-hb975\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-kube-api-access-hb975\") pod \"589f2b68-0b46-4320-880d-ff9e9d047d18\" (UID: \"589f2b68-0b46-4320-880d-ff9e9d047d18\") "
Nov 02 15:53:07.919374 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.919341    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-certs" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:07.919448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.919363    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-data" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:07.919448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.919423    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589f2b68-0b46-4320-880d-ff9e9d047d18-istiod-ca-cert" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:07.919902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.919879    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/589f2b68-0b46-4320-880d-ff9e9d047d18-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.920076 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.920060    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.920193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.920177    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.920297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.920284    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:07.920393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.920020    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-socket" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:07.921232 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.921208    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-podinfo" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:07.921373 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.921356    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-envoy" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:07.921793 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.921769    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-kube-api-access-hb975" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "kube-api-access-hb975". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:07.922351 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:07.922322    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-token" pod "589f2b68-0b46-4320-880d-ff9e9d047d18" (UID: "589f2b68-0b46-4320-880d-ff9e9d047d18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:08.021324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.021280    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:08.021324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.021316    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:08.021324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.021328    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/589f2b68-0b46-4320-880d-ff9e9d047d18-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:08.021589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.021343    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb975\" (UniqueName: \"kubernetes.io/projected/589f2b68-0b46-4320-880d-ff9e9d047d18-kube-api-access-hb975\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:08.021589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.021358    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/589f2b68-0b46-4320-880d-ff9e9d047d18-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:08.222343 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.222309    7019 generic.go:356] "Generic (PLEG): container finished" podID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerID="50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473" exitCode=0
Nov 02 15:53:08.222564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.222366    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"
Nov 02 15:53:08.222564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.222399    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" event={"ID":"589f2b68-0b46-4320-880d-ff9e9d047d18","Type":"ContainerDied","Data":"50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473"}
Nov 02 15:53:08.222564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.222475    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm" event={"ID":"589f2b68-0b46-4320-880d-ff9e9d047d18","Type":"ContainerDied","Data":"59d643f5ee3d5cf32627cd390aee0e469f7d4836162e4e0dd830e765d4054518"}
Nov 02 15:53:08.222564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.222506    7019 scope.go:122] "RemoveContainer" containerID="50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473"
Nov 02 15:53:08.235880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.235862    7019 scope.go:122] "RemoveContainer" containerID="50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473"
Nov 02 15:53:08.236164 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:08.236142    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473\": container with ID starting with 50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473 not found: ID does not exist" containerID="50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473"
Nov 02 15:53:08.236215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.236173    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473"} err="failed to get container status \"50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473\": rpc error: code = NotFound desc = could not find container \"50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473\": container with ID starting with 50127051a25ed7a956eae21e1b9f25c1d846972f60044461517f60e2a0ff9473 not found: ID does not exist"
Nov 02 15:53:08.278494 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.278461    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"]
Nov 02 15:53:08.291244 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.291217    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-fda646-openshift-default-67f4df6b79-jtrrm"]
Nov 02 15:53:08.801582 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:08.801516    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:09.579943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:09.579907    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:09.861553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:09.861524    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" path="/var/lib/kubelet/pods/589f2b68-0b46-4320-880d-ff9e9d047d18/volumes"
Nov 02 15:53:09.914959 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:09.914925    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"]
Nov 02 15:53:10.116385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:10.116289    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:10.392116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:10.391991    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:10.665863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:10.665779    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:10.699761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:10.699721    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"]
Nov 02 15:53:10.700026 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:10.699996    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy" containerID="cri-o://8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b" gracePeriod=30
Nov 02 15:53:10.849447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:10.849409    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:11.112114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.112083    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:11.361017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.360984    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:11.424983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.424891    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"]
Nov 02 15:53:11.437994 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.437958    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a433eb-b2tc2"]
Nov 02 15:53:11.612537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.612492    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"]
Nov 02 15:53:11.613030 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613014    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613030    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613054    7019 cpu_manager.go:414] "containerMap: removing container" podUID="db681668-d7cc-4da3-8835-1b72ef6687f1" containerName="maas-api"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613060    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db681668-d7cc-4da3-8835-1b72ef6687f1" containerName="maas-api"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613072    7019 cpu_manager.go:414] "containerMap: removing container" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613077    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613094    7019 cpu_manager.go:414] "containerMap: removing container" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613098    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613106    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" containerName="revoke-keys"
Nov 02 15:53:11.613108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613112    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" containerName="revoke-keys"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613125    7019 cpu_manager.go:414] "containerMap: removing container" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613130    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613143    7019 cpu_manager.go:414] "containerMap: removing container" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613147    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613213    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="832acbfc-6909-41ec-9aed-b653a7d9a662" containerName="main"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613222    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e394f8f8-2b26-4dcd-8fa9-84a62560e269" containerName="istio-proxy"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613232    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="db681668-d7cc-4da3-8835-1b72ef6687f1" containerName="maas-api"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613239    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="589f2b68-0b46-4320-880d-ff9e9d047d18" containerName="istio-proxy"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613247    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="60cb7c62-9ce5-4a85-96df-00a6ea763e49" containerName="main"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613253    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" containerName="revoke-keys"
Nov 02 15:53:11.613596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.613261    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9a6d0fb-4834-47b4-928f-7baf205291fc" containerName="main"
Nov 02 15:53:11.617810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.617784    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.642044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.641954    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:53:11.642249 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.642050    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:53:11.658761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.658729    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"]
Nov 02 15:53:11.739319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.739219    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:11.775664 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.775603    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbs5h\" (UniqueName: \"kubernetes.io/projected/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-kube-api-access-rbs5h\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.775845 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.775767    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.775908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.775881    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.861440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.861402    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c300c5c6-12ea-41ae-b677-14a3cf4cc0e3" path="/var/lib/kubelet/pods/c300c5c6-12ea-41ae-b677-14a3cf4cc0e3/volumes"
Nov 02 15:53:11.877692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.877634    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbs5h\" (UniqueName: \"kubernetes.io/projected/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-kube-api-access-rbs5h\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.877942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.877733    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.877942 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.877790    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.878362 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.878342    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.880505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.880472    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.908527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.908492    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbs5h\" (UniqueName: \"kubernetes.io/projected/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-kube-api-access-rbs5h\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:11.932127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:11.932103    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:12.250245 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:12.250215    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"]
Nov 02 15:53:12.251592 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:53:12.251563    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e253b69_3b6b_474e_a55c_9e68fe3c2fcf.slice/crio-23aad57d48016cfadfc259acc10261bb54f33b0c1e2055a4ab2a517383b30ce2 WatchSource:0}: Error finding container 23aad57d48016cfadfc259acc10261bb54f33b0c1e2055a4ab2a517383b30ce2: Status 404 returned error can't find the container with id 23aad57d48016cfadfc259acc10261bb54f33b0c1e2055a4ab2a517383b30ce2
Nov 02 15:53:12.305463 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:12.305433    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:13.262436 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:13.262394    7019 generic.go:356] "Generic (PLEG): container finished" podID="3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" containerID="23522528467dba82be2ae8b5d4878eb5ace88c9f7df2f41f1b6495d04db8f7c1" exitCode=0
Nov 02 15:53:13.262870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:13.262434    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb" event={"ID":"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf","Type":"ContainerDied","Data":"23522528467dba82be2ae8b5d4878eb5ace88c9f7df2f41f1b6495d04db8f7c1"}
Nov 02 15:53:13.262870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:13.262493    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb" event={"ID":"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf","Type":"ContainerStarted","Data":"23aad57d48016cfadfc259acc10261bb54f33b0c1e2055a4ab2a517383b30ce2"}
Nov 02 15:53:13.449818 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:13.449701    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:13.916050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:13.916012    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:14.304729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.297484    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:14.545460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.545432    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:14.578855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.578690    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:14.728428 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.728382    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-openshift-ca\") pod \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") "
Nov 02 15:53:14.728676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.728446    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-xks-ca\") pod \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") "
Nov 02 15:53:14.728676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.728549    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-kube-api-access-rbs5h\" (UniqueName: \"kubernetes.io/projected/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-kube-api-access-rbs5h\") pod \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\" (UID: \"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf\") "
Nov 02 15:53:14.729894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.729864    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-openshift-ca" pod "3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" (UID: "3e253b69-3b6b-474e-a55c-9e68fe3c2fcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:14.736468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.735516    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-kube-api-access-rbs5h" pod "3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" (UID: "3e253b69-3b6b-474e-a55c-9e68fe3c2fcf"). InnerVolumeSpecName "kube-api-access-rbs5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:14.736468 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.736027    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-xks-ca" pod "3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" (UID: "3e253b69-3b6b-474e-a55c-9e68fe3c2fcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:14.782667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.777355    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:14.830332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.830233    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:14.830332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.830267    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:14.830332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.830278    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbs5h\" (UniqueName: \"kubernetes.io/projected/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf-kube-api-access-rbs5h\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:14.975663 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:14.975619    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:15.318283 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:15.318244    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"
Nov 02 15:53:15.319240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:15.318363    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb" event={"ID":"3e253b69-3b6b-474e-a55c-9e68fe3c2fcf","Type":"ContainerDied","Data":"23aad57d48016cfadfc259acc10261bb54f33b0c1e2055a4ab2a517383b30ce2"}
Nov 02 15:53:15.319240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:15.319211    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23aad57d48016cfadfc259acc10261bb54f33b0c1e2055a4ab2a517383b30ce2"
Nov 02 15:53:15.358784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:15.358740    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:15.968734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:15.968710    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:53:16.168715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.151149    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-certs\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.169084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.169055    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7d101b-466f-453a-a822-29dc40406ebb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7d101b-466f-453a-a822-29dc40406ebb-istiod-ca-cert\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.169264 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.169248    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-credential-socket\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.169417 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.169403    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-kube-api-access-jw6z8\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-kube-api-access-jw6z8\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.169547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.169535    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-socket\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.169797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.169746    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-envoy\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.169978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.169960    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-istio-token\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-istio-token\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.170105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.170092    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c7d101b-466f-453a-a822-29dc40406ebb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7d101b-466f-453a-a822-29dc40406ebb-istio-podinfo\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.170206 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.170195    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-data\") pod \"9c7d101b-466f-453a-a822-29dc40406ebb\" (UID: \"9c7d101b-466f-453a-a822-29dc40406ebb\") "
Nov 02 15:53:16.176741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.173616    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-credential-socket" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.176741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.173992    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7d101b-466f-453a-a822-29dc40406ebb-istiod-ca-cert" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:16.176741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.174619    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-socket" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.176741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.174847    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-data" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.176741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.175479    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c7d101b-466f-453a-a822-29dc40406ebb-istio-podinfo" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:16.181811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.180049    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.181811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.180079    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c7d101b-466f-453a-a822-29dc40406ebb-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.181811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.180094    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.181811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.180106    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c7d101b-466f-453a-a822-29dc40406ebb-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.181811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.180122    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.181811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.180332    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-istio-token" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:16.181811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.180540    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-kube-api-access-jw6z8" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "kube-api-access-jw6z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:16.182456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.182141    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-envoy" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.182553 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.182526    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-certs" pod "9c7d101b-466f-453a-a822-29dc40406ebb" (UID: "9c7d101b-466f-453a-a822-29dc40406ebb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:16.284590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.281678    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.284590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.281719    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6z8\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-kube-api-access-jw6z8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.284590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.281734    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c7d101b-466f-453a-a822-29dc40406ebb-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.284590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.281750    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c7d101b-466f-453a-a822-29dc40406ebb-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:16.361637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.357246    7019 generic.go:356] "Generic (PLEG): container finished" podID="9c7d101b-466f-453a-a822-29dc40406ebb" containerID="8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b" exitCode=0
Nov 02 15:53:16.361637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.357428    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" event={"ID":"9c7d101b-466f-453a-a822-29dc40406ebb","Type":"ContainerDied","Data":"8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b"}
Nov 02 15:53:16.361637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.357467    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb" event={"ID":"9c7d101b-466f-453a-a822-29dc40406ebb","Type":"ContainerDied","Data":"17ba3a82bd93ee37c7d11ad3a37621f7f921f3386d39df2931d9d7a0c5f07113"}
Nov 02 15:53:16.361637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.357487    7019 scope.go:122] "RemoveContainer" containerID="8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b"
Nov 02 15:53:16.361637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.357602    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"
Nov 02 15:53:16.373935 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.373913    7019 scope.go:122] "RemoveContainer" containerID="8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b"
Nov 02 15:53:16.374980 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:16.374946    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b\": container with ID starting with 8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b not found: ID does not exist" containerID="8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b"
Nov 02 15:53:16.375117 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.374991    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b"} err="failed to get container status \"8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b\": rpc error: code = NotFound desc = could not find container \"8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b\": container with ID starting with 8f3c4b239274ece446741b34f13f67a23b52c99d2bc924592d1201b32f43bc4b not found: ID does not exist"
Nov 02 15:53:16.435716 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.435619    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"]
Nov 02 15:53:16.440327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:16.440291    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-6f2e0c-openshift-default-767db5d9b7-bkfjb"]
Nov 02 15:53:17.134685 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:17.134615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:17.399123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:17.399037    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:17.689590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:17.689505    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:17.867537 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:17.867502    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" path="/var/lib/kubelet/pods/9c7d101b-466f-453a-a822-29dc40406ebb/volumes"
Nov 02 15:53:17.921370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:17.921334    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"]
Nov 02 15:53:17.921602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:17.921579    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy" containerID="cri-o://b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751" gracePeriod=30
Nov 02 15:53:18.025783 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:18.025424    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:19.104222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:19.101337    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:19.980735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:19.980683    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:20.171611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.171578    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:20.265179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.265088    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"]
Nov 02 15:53:20.265379 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.265347    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy" containerID="cri-o://bc58b66f2c4bdd03f49a02f2cea518634de7aae57e2fde07ec137b2dfc8db4bf" gracePeriod=30
Nov 02 15:53:20.336074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.336036    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:20.823585 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.823549    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:20.845639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.845606    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"]
Nov 02 15:53:20.846168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.846149    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" containerName="revoke-keys"
Nov 02 15:53:20.846168 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.846167    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" containerName="revoke-keys"
Nov 02 15:53:20.846295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.846194    7019 cpu_manager.go:414] "containerMap: removing container" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy"
Nov 02 15:53:20.846295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.846200    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy"
Nov 02 15:53:20.846295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.846279    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7d101b-466f-453a-a822-29dc40406ebb" containerName="istio-proxy"
Nov 02 15:53:20.846444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.846296    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" containerName="revoke-keys"
Nov 02 15:53:20.849811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.849794    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:20.872848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.872823    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"]
Nov 02 15:53:20.946867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.946834    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:20.947038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.946873    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gntd\" (UniqueName: \"kubernetes.io/projected/e494ad51-39f7-4e87-9b6c-a24090571b09-kube-api-access-5gntd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:20.947038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.947022    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:20.963415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.963392    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:53:20.963735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:20.963713    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:53:21.048124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.048089    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:21.048304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.048131    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gntd\" (UniqueName: \"kubernetes.io/projected/e494ad51-39f7-4e87-9b6c-a24090571b09-kube-api-access-5gntd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:21.048304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.048263    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:21.048980 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.048952    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:21.050552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.050530    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:21.073218 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.073189    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gntd\" (UniqueName: \"kubernetes.io/projected/e494ad51-39f7-4e87-9b6c-a24090571b09-kube-api-access-5gntd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:21.163821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.163789    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:21.480290 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:53:21.480168    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode494ad51_39f7_4e87_9b6c_a24090571b09.slice/crio-2689b074666110ea3cdd922458e57ca9bd6dd8bb19214c983335c8ffc817e49b WatchSource:0}: Error finding container 2689b074666110ea3cdd922458e57ca9bd6dd8bb19214c983335c8ffc817e49b: Status 404 returned error can't find the container with id 2689b074666110ea3cdd922458e57ca9bd6dd8bb19214c983335c8ffc817e49b
Nov 02 15:53:21.480707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:21.480563    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"]
Nov 02 15:53:22.250591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:22.250512    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:22.433181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:22.433142    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6" event={"ID":"e494ad51-39f7-4e87-9b6c-a24090571b09","Type":"ContainerStarted","Data":"661f39b386c8df807b972580d2728c2f60a6e72556ad209b9e81332a626887f5"}
Nov 02 15:53:22.433361 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:22.433213    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6" event={"ID":"e494ad51-39f7-4e87-9b6c-a24090571b09","Type":"ContainerStarted","Data":"2689b074666110ea3cdd922458e57ca9bd6dd8bb19214c983335c8ffc817e49b"}
Nov 02 15:53:22.499735 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:22.499638    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6" podStartSLOduration=2.499614752 podStartE2EDuration="2.499614752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:53:22.486749787 +0000 UTC m=+3396.740703412" watchObservedRunningTime="2026-11-02 15:53:22.499614752 +0000 UTC m=+3396.753568376"
Nov 02 15:53:22.514933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:22.514900    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:22.772034 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:22.771948    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:53:22.839698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:22.839659    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:23.197342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.197315    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:53:23.282856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.282826    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-token\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-token\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.283023 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.282963    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istiod-ca-cert\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.283112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283007    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-data\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.283112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283078    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-envoy\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.283409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283139    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-credential-socket\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.283546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283372    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-data" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:23.283608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283553    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-podinfo\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.283683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283603    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-credential-socket" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:23.283683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283618    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-kube-api-access-hz2br\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-kube-api-access-hz2br\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.284177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.283393    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istiod-ca-cert" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:23.284519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.284391    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-certs\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.284938 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.284898    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-certs" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:23.285302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.285070    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-socket\") pod \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\" (UID: \"d7d403ee-1f73-4b4a-bcb6-85833d42e737\") "
Nov 02 15:53:23.285482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.285456    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-socket" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:23.286011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.285986    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.286109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286012    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.286109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286026    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.286109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286037    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.286109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286050    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.286429 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286405    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-token" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:23.286738 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286714    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-envoy" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:23.286822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286740    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-podinfo" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:23.286822 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.286793    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-kube-api-access-hz2br" pod "d7d403ee-1f73-4b4a-bcb6-85833d42e737" (UID: "d7d403ee-1f73-4b4a-bcb6-85833d42e737"). InnerVolumeSpecName "kube-api-access-hz2br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:23.392541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.387737    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.392541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.387776    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.392541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.387792    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d403ee-1f73-4b4a-bcb6-85833d42e737-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.392541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.387827    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz2br\" (UniqueName: \"kubernetes.io/projected/d7d403ee-1f73-4b4a-bcb6-85833d42e737-kube-api-access-hz2br\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:23.426877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.426831    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:23.443055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.443021    7019 generic.go:356] "Generic (PLEG): container finished" podID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerID="b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751" exitCode=0
Nov 02 15:53:23.443215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.443102    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" event={"ID":"d7d403ee-1f73-4b4a-bcb6-85833d42e737","Type":"ContainerDied","Data":"b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751"}
Nov 02 15:53:23.443215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.443113    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"
Nov 02 15:53:23.443215 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.443189    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z" event={"ID":"d7d403ee-1f73-4b4a-bcb6-85833d42e737","Type":"ContainerDied","Data":"e99e079e4469441ca030f26d170462a9cf3963c5214db7be37fe989cace1478a"}
Nov 02 15:53:23.443399 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.443221    7019 scope.go:122] "RemoveContainer" containerID="b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751"
Nov 02 15:53:23.445452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.445426    7019 generic.go:356] "Generic (PLEG): container finished" podID="e494ad51-39f7-4e87-9b6c-a24090571b09" containerID="661f39b386c8df807b972580d2728c2f60a6e72556ad209b9e81332a626887f5" exitCode=0
Nov 02 15:53:23.445593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.445512    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6" event={"ID":"e494ad51-39f7-4e87-9b6c-a24090571b09","Type":"ContainerDied","Data":"661f39b386c8df807b972580d2728c2f60a6e72556ad209b9e81332a626887f5"}
Nov 02 15:53:23.455913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.455892    7019 scope.go:122] "RemoveContainer" containerID="b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751"
Nov 02 15:53:23.456186 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:23.456168    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751\": container with ID starting with b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751 not found: ID does not exist" containerID="b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751"
Nov 02 15:53:23.456248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.456194    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751"} err="failed to get container status \"b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751\": rpc error: code = NotFound desc = could not find container \"b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751\": container with ID starting with b893c416e26e404265e8cee856eb53260b12cfeb5082b6d552a078e5aef36751 not found: ID does not exist"
Nov 02 15:53:23.504766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.504734    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"]
Nov 02 15:53:23.510477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.510446    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6af50f-openshift-default-8545fb68f-j427z"]
Nov 02 15:53:23.844688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.844635    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:23.863118 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.863089    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" path="/var/lib/kubelet/pods/d7d403ee-1f73-4b4a-bcb6-85833d42e737/volumes"
Nov 02 15:53:23.948301 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.948262    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"]
Nov 02 15:53:23.951214 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:23.951183    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-797488-fjqcb"]
Nov 02 15:53:24.012278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.012240    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:24.606420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.606396    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:24.702985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.702947    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-xks-ca\") pod \"e494ad51-39f7-4e87-9b6c-a24090571b09\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") "
Nov 02 15:53:24.703164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.703065    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-openshift-ca\") pod \"e494ad51-39f7-4e87-9b6c-a24090571b09\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") "
Nov 02 15:53:24.703265 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.703169    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e494ad51-39f7-4e87-9b6c-a24090571b09-kube-api-access-5gntd\" (UniqueName: \"kubernetes.io/projected/e494ad51-39f7-4e87-9b6c-a24090571b09-kube-api-access-5gntd\") pod \"e494ad51-39f7-4e87-9b6c-a24090571b09\" (UID: \"e494ad51-39f7-4e87-9b6c-a24090571b09\") "
Nov 02 15:53:24.703449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.703421    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-openshift-ca" pod "e494ad51-39f7-4e87-9b6c-a24090571b09" (UID: "e494ad51-39f7-4e87-9b6c-a24090571b09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:24.703723 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.703705    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:24.705143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.705088    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-xks-ca" pod "e494ad51-39f7-4e87-9b6c-a24090571b09" (UID: "e494ad51-39f7-4e87-9b6c-a24090571b09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:24.705243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.705151    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e494ad51-39f7-4e87-9b6c-a24090571b09-kube-api-access-5gntd" pod "e494ad51-39f7-4e87-9b6c-a24090571b09" (UID: "e494ad51-39f7-4e87-9b6c-a24090571b09"). InnerVolumeSpecName "kube-api-access-5gntd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:24.804581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.804545    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e494ad51-39f7-4e87-9b6c-a24090571b09-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:24.804581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.804575    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gntd\" (UniqueName: \"kubernetes.io/projected/e494ad51-39f7-4e87-9b6c-a24090571b09-kube-api-access-5gntd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:24.850742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.850705    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:24.935601 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:24.935563    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx" podUID="cc772fc1-9d2d-4177-b91b-a6ebdf72b920" containerName="main" containerID="cri-o://d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059" gracePeriod=45
Nov 02 15:53:25.181979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.181946    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:53:25.307222 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.307102    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:25.313158 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313131    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tmp-dir\") pod \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") "
Nov 02 15:53:25.313307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313194    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-home\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-home\") pod \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") "
Nov 02 15:53:25.313307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313289    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-dshm\") pod \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") "
Nov 02 15:53:25.313433 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313359    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tls-certs\") pod \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") "
Nov 02 15:53:25.313498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313426    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-kube-api-access-45dlk\" (UniqueName: \"kubernetes.io/projected/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-kube-api-access-45dlk\") pod \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") "
Nov 02 15:53:25.313498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313416    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tmp-dir" pod "cc772fc1-9d2d-4177-b91b-a6ebdf72b920" (UID: "cc772fc1-9d2d-4177-b91b-a6ebdf72b920"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.313498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313483    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-model-cache\") pod \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\" (UID: \"cc772fc1-9d2d-4177-b91b-a6ebdf72b920\") "
Nov 02 15:53:25.313701 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313497    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-home" pod "cc772fc1-9d2d-4177-b91b-a6ebdf72b920" (UID: "cc772fc1-9d2d-4177-b91b-a6ebdf72b920"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.313857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.313823    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-model-cache" pod "cc772fc1-9d2d-4177-b91b-a6ebdf72b920" (UID: "cc772fc1-9d2d-4177-b91b-a6ebdf72b920"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.314029 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.314010    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.314103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.314036    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.314103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.314052    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.315625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.315601    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-kube-api-access-45dlk" pod "cc772fc1-9d2d-4177-b91b-a6ebdf72b920" (UID: "cc772fc1-9d2d-4177-b91b-a6ebdf72b920"). InnerVolumeSpecName "kube-api-access-45dlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:25.315747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.315624    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tls-certs" pod "cc772fc1-9d2d-4177-b91b-a6ebdf72b920" (UID: "cc772fc1-9d2d-4177-b91b-a6ebdf72b920"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:25.315747 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.315610    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-dshm" pod "cc772fc1-9d2d-4177-b91b-a6ebdf72b920" (UID: "cc772fc1-9d2d-4177-b91b-a6ebdf72b920"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.414971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.414927    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45dlk\" (UniqueName: \"kubernetes.io/projected/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-kube-api-access-45dlk\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.414971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.414959    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.414971 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.414968    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc772fc1-9d2d-4177-b91b-a6ebdf72b920-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.465001 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.464964    7019 generic.go:356] "Generic (PLEG): container finished" podID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerID="bc58b66f2c4bdd03f49a02f2cea518634de7aae57e2fde07ec137b2dfc8db4bf" exitCode=0
Nov 02 15:53:25.465213 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.465188    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" event={"ID":"89d220ea-b8e7-492d-9036-68c13c08ece9","Type":"ContainerDied","Data":"bc58b66f2c4bdd03f49a02f2cea518634de7aae57e2fde07ec137b2dfc8db4bf"}
Nov 02 15:53:25.465297 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.465258    7019 scope.go:122] "RemoveContainer" containerID="e624d8538a0cfe9baaf4e3dac96c76b088720eac9e10c8ce91032d62d5befc34"
Nov 02 15:53:25.467413 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.467386    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6" event={"ID":"e494ad51-39f7-4e87-9b6c-a24090571b09","Type":"ContainerDied","Data":"2689b074666110ea3cdd922458e57ca9bd6dd8bb19214c983335c8ffc817e49b"}
Nov 02 15:53:25.467536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.467441    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2689b074666110ea3cdd922458e57ca9bd6dd8bb19214c983335c8ffc817e49b"
Nov 02 15:53:25.467536 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.467399    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"
Nov 02 15:53:25.472094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.470908    7019 generic.go:356] "Generic (PLEG): container finished" podID="cc772fc1-9d2d-4177-b91b-a6ebdf72b920" containerID="d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059" exitCode=0
Nov 02 15:53:25.472094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.471039    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx" event={"ID":"cc772fc1-9d2d-4177-b91b-a6ebdf72b920","Type":"ContainerDied","Data":"d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059"}
Nov 02 15:53:25.472094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.471077    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx" event={"ID":"cc772fc1-9d2d-4177-b91b-a6ebdf72b920","Type":"ContainerDied","Data":"19d55d463d392075eb01b3b483b3f3e74a52d19ee8b6b5d9f0046fc2d2458c65"}
Nov 02 15:53:25.472094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.471135    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"
Nov 02 15:53:25.494477 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.494453    7019 scope.go:122] "RemoveContainer" containerID="d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059"
Nov 02 15:53:25.507617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.507594    7019 scope.go:122] "RemoveContainer" containerID="d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059"
Nov 02 15:53:25.508218 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:25.508189    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059\": container with ID starting with d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059 not found: ID does not exist" containerID="d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059"
Nov 02 15:53:25.508322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.508229    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059"} err="failed to get container status \"d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059\": rpc error: code = NotFound desc = could not find container \"d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059\": container with ID starting with d891583c240ca33c815daa57f0a4d3f4a7a1531e5696e7a7e68a74ac1e590059 not found: ID does not exist"
Nov 02 15:53:25.531595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.531570    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:53:25.552155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.552127    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"]
Nov 02 15:53:25.556501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.556476    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f0039c/ipp-route-f0039c-kserve-6b66d8db85-gnkzx"]
Nov 02 15:53:25.617448 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617418    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-kube-api-access-v9klv\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-kube-api-access-v9klv\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.617933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617485    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-data\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.617933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617537    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-podinfo\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.617933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617574    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-socket\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.617933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617768    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-token\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-token\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.617933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617869    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-data" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.617933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617912    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-envoy\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.617933 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617916    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-socket" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.618455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617950    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-certs\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.618455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.617984    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d220ea-b8e7-492d-9036-68c13c08ece9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d220ea-b8e7-492d-9036-68c13c08ece9-istiod-ca-cert\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.618455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618062    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-credential-socket\") pod \"89d220ea-b8e7-492d-9036-68c13c08ece9\" (UID: \"89d220ea-b8e7-492d-9036-68c13c08ece9\") "
Nov 02 15:53:25.618455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618217    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-certs" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.618455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618416    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-credential-socket" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.618455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618443    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d220ea-b8e7-492d-9036-68c13c08ece9-istiod-ca-cert" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:25.618833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618574    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.618833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618592    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d220ea-b8e7-492d-9036-68c13c08ece9-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.618833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618605    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.618833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618620    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.618833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.618634    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.620475 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.620440    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-token" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:25.620874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.620715    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-envoy" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:25.620999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.620876    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-podinfo" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:25.620999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.620916    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-kube-api-access-v9klv" pod "89d220ea-b8e7-492d-9036-68c13c08ece9" (UID: "89d220ea-b8e7-492d-9036-68c13c08ece9"). InnerVolumeSpecName "kube-api-access-v9klv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:25.720089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.719998    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.720089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.720033    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.720089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.720042    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9klv\" (UniqueName: \"kubernetes.io/projected/89d220ea-b8e7-492d-9036-68c13c08ece9-kube-api-access-v9klv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.720089 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.720053    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d220ea-b8e7-492d-9036-68c13c08ece9-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:25.737101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.737066    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:25.861865 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.861842    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e253b69-3b6b-474e-a55c-9e68fe3c2fcf" path="/var/lib/kubelet/pods/3e253b69-3b6b-474e-a55c-9e68fe3c2fcf/volumes"
Nov 02 15:53:25.862330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:25.862314    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc772fc1-9d2d-4177-b91b-a6ebdf72b920" path="/var/lib/kubelet/pods/cc772fc1-9d2d-4177-b91b-a6ebdf72b920/volumes"
Nov 02 15:53:26.491385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:26.491326    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b" event={"ID":"89d220ea-b8e7-492d-9036-68c13c08ece9","Type":"ContainerDied","Data":"4791a49ecc648533a791a85fa8f13de3497a25385f936e9d9c1a22e55a410302"}
Nov 02 15:53:26.491587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:26.491430    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"
Nov 02 15:53:26.491587 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:26.491508    7019 scope.go:122] "RemoveContainer" containerID="bc58b66f2c4bdd03f49a02f2cea518634de7aae57e2fde07ec137b2dfc8db4bf"
Nov 02 15:53:26.560911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:26.560879    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"]
Nov 02 15:53:26.566398 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:26.566369    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a433eb-openshift-default-7954f9d789-wbp6b"]
Nov 02 15:53:27.362280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:27.362245    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:27.861965 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:27.861938    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" path="/var/lib/kubelet/pods/89d220ea-b8e7-492d-9036-68c13c08ece9/volumes"
Nov 02 15:53:28.382675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:28.381607    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:53:28.898898 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:28.898854    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:30.720880 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:30.720836    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:31.338319 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:31.338276    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"]
Nov 02 15:53:31.499137 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:31.499102    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:31.720149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:31.720117    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:32.529611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:32.529574    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:33.152170 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:33.152138    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:33.391396 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:33.391129    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:33.446885 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:33.446810    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"]
Nov 02 15:53:33.447061 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:33.447037    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" containerID="cri-o://7e590cf1ca2ae9df7d72225431ea5b2e785d68c3a65c222a2180f06f8a263d13" gracePeriod=30
Nov 02 15:53:33.611960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:33.611923    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:34.018481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.018443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:34.193525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.193493    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"]
Nov 02 15:53:34.194057 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194039    7019 cpu_manager.go:414] "containerMap: removing container" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy"
Nov 02 15:53:34.194105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194060    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy"
Nov 02 15:53:34.194105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194086    7019 cpu_manager.go:414] "containerMap: removing container" podUID="cc772fc1-9d2d-4177-b91b-a6ebdf72b920" containerName="main"
Nov 02 15:53:34.194105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194092    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc772fc1-9d2d-4177-b91b-a6ebdf72b920" containerName="main"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194116    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e494ad51-39f7-4e87-9b6c-a24090571b09" containerName="revoke-keys"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194122    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e494ad51-39f7-4e87-9b6c-a24090571b09" containerName="revoke-keys"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194141    7019 cpu_manager.go:414] "containerMap: removing container" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194147    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194220    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e494ad51-39f7-4e87-9b6c-a24090571b09" containerName="revoke-keys"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194230    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194236    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d403ee-1f73-4b4a-bcb6-85833d42e737" containerName="istio-proxy"
Nov 02 15:53:34.194240 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194243    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy"
Nov 02 15:53:34.194517 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.194254    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc772fc1-9d2d-4177-b91b-a6ebdf72b920" containerName="main"
Nov 02 15:53:34.199408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.199385    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.221903 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.221873    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:53:34.234821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.234794    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:53:34.239101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.239072    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"]
Nov 02 15:53:34.343888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.343787    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.343888 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.343854    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.344147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.343925    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jql9j\" (UniqueName: \"kubernetes.io/projected/33c87b12-0c06-4d00-996a-59978391e30f-kube-api-access-jql9j\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.380044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.380003    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:34.445731 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.445695    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jql9j\" (UniqueName: \"kubernetes.io/projected/33c87b12-0c06-4d00-996a-59978391e30f-kube-api-access-jql9j\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.445926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.445822    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.445926 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.445860    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.446539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.446517    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.448242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.448221    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.476759 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.476728    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jql9j\" (UniqueName: \"kubernetes.io/projected/33c87b12-0c06-4d00-996a-59978391e30f-kube-api-access-jql9j\") pod \"maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.513593 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.513561    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:34.831629 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:53:34.831595    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c87b12_0c06_4d00_996a_59978391e30f.slice/crio-ba1665cd2169f47ccdd72db723c756c7b4b155c01646facfee745bc82cc25e0a WatchSource:0}: Error finding container ba1665cd2169f47ccdd72db723c756c7b4b155c01646facfee745bc82cc25e0a: Status 404 returned error can't find the container with id ba1665cd2169f47ccdd72db723c756c7b4b155c01646facfee745bc82cc25e0a
Nov 02 15:53:34.832133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:34.831958    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"]
Nov 02 15:53:35.617853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:35.617817    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q" event={"ID":"33c87b12-0c06-4d00-996a-59978391e30f","Type":"ContainerStarted","Data":"3e436749bdb198f015343782fbd23dc643334eac0f55a14cec27ffd9e8a31d0e"}
Nov 02 15:53:35.618013 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:35.617874    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q" event={"ID":"33c87b12-0c06-4d00-996a-59978391e30f","Type":"ContainerStarted","Data":"ba1665cd2169f47ccdd72db723c756c7b4b155c01646facfee745bc82cc25e0a"}
Nov 02 15:53:35.675307 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:35.675250    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q" podStartSLOduration=1.675236771 podStartE2EDuration="1.675236771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:53:35.671677194 +0000 UTC m=+3409.925630815" watchObservedRunningTime="2026-11-02 15:53:35.675236771 +0000 UTC m=+3409.929190395"
Nov 02 15:53:36.094904 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.094811    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:36.338561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.338515    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" podUID="0418a181-53a9-43a2-b261-dac04482277f" containerName="payload-pre-processing" containerID="cri-o://9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613" gracePeriod=25
Nov 02 15:53:36.628305 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.628267    7019 generic.go:356] "Generic (PLEG): container finished" podID="33c87b12-0c06-4d00-996a-59978391e30f" containerID="3e436749bdb198f015343782fbd23dc643334eac0f55a14cec27ffd9e8a31d0e" exitCode=0
Nov 02 15:53:36.628491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.628343    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q" event={"ID":"33c87b12-0c06-4d00-996a-59978391e30f","Type":"ContainerDied","Data":"3e436749bdb198f015343782fbd23dc643334eac0f55a14cec27ffd9e8a31d0e"}
Nov 02 15:53:36.655185 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.655148    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:36.745823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.745707    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"]
Nov 02 15:53:36.747707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.747641    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f6f1a0-4ppj6"]
Nov 02 15:53:36.899380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:36.899284    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:37.319896 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.319801    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:37.569449 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.569411    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:53:37.569627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.569468    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:53:37.798805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.798776    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:37.863564 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.863531    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e494ad51-39f7-4e87-9b6c-a24090571b09" path="/var/lib/kubelet/pods/e494ad51-39f7-4e87-9b6c-a24090571b09/volumes"
Nov 02 15:53:37.887115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.887029    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c87b12-0c06-4d00-996a-59978391e30f-kube-api-access-jql9j\" (UniqueName: \"kubernetes.io/projected/33c87b12-0c06-4d00-996a-59978391e30f-kube-api-access-jql9j\") pod \"33c87b12-0c06-4d00-996a-59978391e30f\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") "
Nov 02 15:53:37.887295 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.887120    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-openshift-ca\") pod \"33c87b12-0c06-4d00-996a-59978391e30f\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") "
Nov 02 15:53:37.887378 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.887295    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-xks-ca\") pod \"33c87b12-0c06-4d00-996a-59978391e30f\" (UID: \"33c87b12-0c06-4d00-996a-59978391e30f\") "
Nov 02 15:53:37.887495 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.887459    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-openshift-ca" pod "33c87b12-0c06-4d00-996a-59978391e30f" (UID: "33c87b12-0c06-4d00-996a-59978391e30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:37.887706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.887685    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:37.889354 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.889326    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-xks-ca" pod "33c87b12-0c06-4d00-996a-59978391e30f" (UID: "33c87b12-0c06-4d00-996a-59978391e30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:37.889497 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.889395    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c87b12-0c06-4d00-996a-59978391e30f-kube-api-access-jql9j" pod "33c87b12-0c06-4d00-996a-59978391e30f" (UID: "33c87b12-0c06-4d00-996a-59978391e30f"). InnerVolumeSpecName "kube-api-access-jql9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:37.989075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.989031    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c87b12-0c06-4d00-996a-59978391e30f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:37.989075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:37.989077    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jql9j\" (UniqueName: \"kubernetes.io/projected/33c87b12-0c06-4d00-996a-59978391e30f-kube-api-access-jql9j\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.646715 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.646679    7019 generic.go:356] "Generic (PLEG): container finished" podID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerID="7e590cf1ca2ae9df7d72225431ea5b2e785d68c3a65c222a2180f06f8a263d13" exitCode=0
Nov 02 15:53:38.647186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.646766    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" event={"ID":"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf","Type":"ContainerDied","Data":"7e590cf1ca2ae9df7d72225431ea5b2e785d68c3a65c222a2180f06f8a263d13"}
Nov 02 15:53:38.647186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.646839    7019 scope.go:122] "RemoveContainer" containerID="d7fd9341859c315bf8a6ff50b0c28a7c84cb373ef80398ef7df0f0a1abc8de41"
Nov 02 15:53:38.648907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.648550    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"
Nov 02 15:53:38.648907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.648580    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q" event={"ID":"33c87b12-0c06-4d00-996a-59978391e30f","Type":"ContainerDied","Data":"ba1665cd2169f47ccdd72db723c756c7b4b155c01646facfee745bc82cc25e0a"}
Nov 02 15:53:38.648907 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.648686    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1665cd2169f47ccdd72db723c756c7b4b155c01646facfee745bc82cc25e0a"
Nov 02 15:53:38.723323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.723294    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:53:38.873554 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.873506    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:38.899511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899478    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-kube-api-access-hnbvq\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-kube-api-access-hnbvq\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.899706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899540    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-credential-socket\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.899706 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899679    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-envoy\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.899857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899714    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-certs\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.899857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899755    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-socket\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.899857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899794    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istiod-ca-cert\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.899857 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899836    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-data\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.900091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899877    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-podinfo\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.900091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900014    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-token\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-token\") pod \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\" (UID: \"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf\") "
Nov 02 15:53:38.900091 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899830    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-credential-socket" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.900261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899947    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-certs" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.900261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.899977    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-socket" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.900261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900081    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-data" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.900261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900148    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istiod-ca-cert" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:38.900464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900401    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.900464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900417    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.900464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900432    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.900464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900445    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.900464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.900459    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:38.903004 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.902975    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-token" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:38.903148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.903028    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-envoy" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:53:38.903148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.903045    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-podinfo" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:53:38.903148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:38.903117    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-kube-api-access-hnbvq" pod "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" (UID: "4cd4992d-7fc2-4a73-a452-9aa7de71aeaf"). InnerVolumeSpecName "kube-api-access-hnbvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:39.001571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.001473    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbvq\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-kube-api-access-hnbvq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:39.001571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.001503    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:39.001571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.001512    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:39.001571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.001520    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:39.049134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.049101    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:39.657331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.657302    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"
Nov 02 15:53:39.657807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.657345    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq" event={"ID":"4cd4992d-7fc2-4a73-a452-9aa7de71aeaf","Type":"ContainerDied","Data":"999a4b4ca693a0bc9a2958ab58510b14839a8ed2bd4f423ee9e5839bbd8643ef"}
Nov 02 15:53:39.657807 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.657404    7019 scope.go:122] "RemoveContainer" containerID="7e590cf1ca2ae9df7d72225431ea5b2e785d68c3a65c222a2180f06f8a263d13"
Nov 02 15:53:39.727547 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.727488    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"]
Nov 02 15:53:39.740274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.740233    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-797488-openshift-default-5f79c8745f-tbjbq"]
Nov 02 15:53:39.861757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:39.861723    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" path="/var/lib/kubelet/pods/4cd4992d-7fc2-4a73-a452-9aa7de71aeaf/volumes"
Nov 02 15:53:41.038608 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:41.038571    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:42.206568 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:42.206524    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:53:42.788139 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:42.788104    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:43.279952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:43.279909    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" podUID="0418a181-53a9-43a2-b261-dac04482277f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:44.669560 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:44.669520    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:44.927941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:44.927840    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:45.120060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:45.120027    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:45.338456 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:45.338369    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:45.624702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:45.620869    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:46.199899 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.199858    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"]
Nov 02 15:53:46.200341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200327    7019 cpu_manager.go:414] "containerMap: removing container" podUID="33c87b12-0c06-4d00-996a-59978391e30f" containerName="revoke-keys"
Nov 02 15:53:46.200391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200342    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c87b12-0c06-4d00-996a-59978391e30f" containerName="revoke-keys"
Nov 02 15:53:46.200391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200355    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy"
Nov 02 15:53:46.200391 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200360    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy"
Nov 02 15:53:46.200490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200401    7019 cpu_manager.go:414] "containerMap: removing container" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy"
Nov 02 15:53:46.200490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200407    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d220ea-b8e7-492d-9036-68c13c08ece9" containerName="istio-proxy"
Nov 02 15:53:46.200490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200480    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy"
Nov 02 15:53:46.200627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200492    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c87b12-0c06-4d00-996a-59978391e30f" containerName="revoke-keys"
Nov 02 15:53:46.200627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.200498    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy"
Nov 02 15:53:46.205395 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.205375    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.222921 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.222895    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:53:46.233883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.233838    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:53:46.236400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.236373    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"]
Nov 02 15:53:46.288998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.288964    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzgvp\" (UniqueName: \"kubernetes.io/projected/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-kube-api-access-nzgvp\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.289138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.289005    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.289138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.289077    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.390235 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.390197    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.390407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.390372    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzgvp\" (UniqueName: \"kubernetes.io/projected/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-kube-api-access-nzgvp\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.390407 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.390392    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.391133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.391107    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.392532 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.392506    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.419046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.419022    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzgvp\" (UniqueName: \"kubernetes.io/projected/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-kube-api-access-nzgvp\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.519043 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.518972    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:46.640323 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.640270    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:46.843086 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:46.843058    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"]
Nov 02 15:53:46.844156 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:53:46.844124    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329889be_c09f_4ad4_bcbc_c8fdb63e1dcc.slice/crio-e3015a943472242712a46a1f4da0c291c11e9da766ebba7907ae3593a4d4b66b WatchSource:0}: Error finding container e3015a943472242712a46a1f4da0c291c11e9da766ebba7907ae3593a4d4b66b: Status 404 returned error can't find the container with id e3015a943472242712a46a1f4da0c291c11e9da766ebba7907ae3593a4d4b66b
Nov 02 15:53:47.718660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:47.718605    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m" event={"ID":"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc","Type":"ContainerStarted","Data":"7b961fc329b35ce7fac3fb0027dd9d5ddc0c2e4c580aa19a6ca66f503cb4b2e4"}
Nov 02 15:53:47.719093 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:47.718698    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m" event={"ID":"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc","Type":"ContainerStarted","Data":"e3015a943472242712a46a1f4da0c291c11e9da766ebba7907ae3593a4d4b66b"}
Nov 02 15:53:47.774237 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:47.774175    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m" podStartSLOduration=1.77415659 podStartE2EDuration="1.77415659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:53:47.772906189 +0000 UTC m=+3422.026859814" watchObservedRunningTime="2026-11-02 15:53:47.77415659 +0000 UTC m=+3422.028110215"
Nov 02 15:53:48.648124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:48.648076    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:48.739068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:48.739028    7019 generic.go:356] "Generic (PLEG): container finished" podID="329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" containerID="7b961fc329b35ce7fac3fb0027dd9d5ddc0c2e4c580aa19a6ca66f503cb4b2e4" exitCode=0
Nov 02 15:53:48.739590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:48.739117    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m" event={"ID":"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc","Type":"ContainerDied","Data":"7b961fc329b35ce7fac3fb0027dd9d5ddc0c2e4c580aa19a6ca66f503cb4b2e4"}
Nov 02 15:53:48.819567 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:48.819530    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:49.894270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:49.894246    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:49.930925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:49.930899    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-xks-ca\") pod \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") "
Nov 02 15:53:49.931065 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:49.931050    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-kube-api-access-nzgvp\" (UniqueName: \"kubernetes.io/projected/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-kube-api-access-nzgvp\") pod \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") "
Nov 02 15:53:49.931184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:49.931169    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-openshift-ca\") pod \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\" (UID: \"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc\") "
Nov 02 15:53:49.931519 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:49.931491    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-openshift-ca" pod "329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" (UID: "329889be-c09f-4ad4-bcbc-c8fdb63e1dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:49.933127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:49.933104    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-xks-ca" pod "329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" (UID: "329889be-c09f-4ad4-bcbc-c8fdb63e1dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:53:49.933201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:49.933137    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-kube-api-access-nzgvp" pod "329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" (UID: "329889be-c09f-4ad4-bcbc-c8fdb63e1dcc"). InnerVolumeSpecName "kube-api-access-nzgvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:50.032683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.032582    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzgvp\" (UniqueName: \"kubernetes.io/projected/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-kube-api-access-nzgvp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.032683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.032614    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.032683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.032626    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:50.042905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.042876    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:50.419111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.418891    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:50.754620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.754537    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m" event={"ID":"329889be-c09f-4ad4-bcbc-c8fdb63e1dcc","Type":"ContainerDied","Data":"e3015a943472242712a46a1f4da0c291c11e9da766ebba7907ae3593a4d4b66b"}
Nov 02 15:53:50.754620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.754593    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3015a943472242712a46a1f4da0c291c11e9da766ebba7907ae3593a4d4b66b"
Nov 02 15:53:50.754620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:50.754554    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"
Nov 02 15:53:51.010278 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:51.010194    7019 scope.go:122] "RemoveContainer" containerID="6d6b87b8cc8c9cd152c2d9b0f0dd76fa98bc876ff65d9102eb0c5ec45b30e1c6"
Nov 02 15:53:51.020714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:51.020693    7019 scope.go:122] "RemoveContainer" containerID="b7e90013e0bc53b822c9668e302ce7c6387c2567515d945f6eb6b55191faa04e"
Nov 02 15:53:51.029960 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:51.029939    7019 scope.go:122] "RemoveContainer" containerID="cefcdfee4b43a6fc333f570ae704fca0f9ffc2afb4394c2c5c20162e8d39228c"
Nov 02 15:53:51.038740 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:51.038715    7019 scope.go:122] "RemoveContainer" containerID="9ab755672ea89618021a603763cb9b677bbd2d1b2d5452a37e5ba4447e4b1fea"
Nov 02 15:53:51.050151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:51.049075    7019 scope.go:122] "RemoveContainer" containerID="23eed0602d745c7570352b43ae31029daa8b2f864da7d17c6abdb662119189be"
Nov 02 15:53:51.059112 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:51.059094    7019 scope.go:122] "RemoveContainer" containerID="8ec257e3628bd0598c10ef934e39245e404a6e1b59d01e5c44acb94f88307302"
Nov 02 15:53:51.067724 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:51.067701    7019 scope.go:122] "RemoveContainer" containerID="815b4a47af0b0433d7573e34a7f8f0493d5a51a6da536fa5c7808b48321275ea"
Nov 02 15:53:52.123671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:52.118213    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:52.706487 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:52.706442    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:53:53.279677 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:53.279607    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" podUID="0418a181-53a9-43a2-b261-dac04482277f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:53:53.933024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:53.932987    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:56.138499 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.138464    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:56.286389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.286344    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:56.601991 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.601966    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:53:56.712452 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.712374    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0418a181-53a9-43a2-b261-dac04482277f-kube-api-access-w56h7\" (UniqueName: \"kubernetes.io/projected/0418a181-53a9-43a2-b261-dac04482277f-kube-api-access-w56h7\") pod \"0418a181-53a9-43a2-b261-dac04482277f\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") "
Nov 02 15:53:56.712611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.712472    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0418a181-53a9-43a2-b261-dac04482277f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0418a181-53a9-43a2-b261-dac04482277f-plugins-config-volume\") pod \"0418a181-53a9-43a2-b261-dac04482277f\" (UID: \"0418a181-53a9-43a2-b261-dac04482277f\") "
Nov 02 15:53:56.712956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.712927    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0418a181-53a9-43a2-b261-dac04482277f-plugins-config-volume" pod "0418a181-53a9-43a2-b261-dac04482277f" (UID: "0418a181-53a9-43a2-b261-dac04482277f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:53:56.714683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.714637    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0418a181-53a9-43a2-b261-dac04482277f-kube-api-access-w56h7" pod "0418a181-53a9-43a2-b261-dac04482277f" (UID: "0418a181-53a9-43a2-b261-dac04482277f"). InnerVolumeSpecName "kube-api-access-w56h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:53:56.805045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.804706    7019 generic.go:356] "Generic (PLEG): container finished" podID="0418a181-53a9-43a2-b261-dac04482277f" containerID="9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613" exitCode=0
Nov 02 15:53:56.805045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.804757    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" event={"ID":"0418a181-53a9-43a2-b261-dac04482277f","Type":"ContainerDied","Data":"9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613"}
Nov 02 15:53:56.805045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.804766    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"
Nov 02 15:53:56.805045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.804831    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75" event={"ID":"0418a181-53a9-43a2-b261-dac04482277f","Type":"ContainerDied","Data":"cc9dc23e184b5efa43c332af44a2df8dcb7a725b7cac38373fd874128611ed25"}
Nov 02 15:53:56.805045 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.804876    7019 scope.go:122] "RemoveContainer" containerID="9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613"
Nov 02 15:53:56.815242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.815212    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56h7\" (UniqueName: \"kubernetes.io/projected/0418a181-53a9-43a2-b261-dac04482277f-kube-api-access-w56h7\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:56.815412 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.815244    7019 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0418a181-53a9-43a2-b261-dac04482277f-plugins-config-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:53:56.826693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.826637    7019 scope.go:122] "RemoveContainer" containerID="9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613"
Nov 02 15:53:56.827044 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:53:56.827022    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613\": container with ID starting with 9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613 not found: ID does not exist" containerID="9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613"
Nov 02 15:53:56.827116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.827058    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613"} err="failed to get container status \"9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613\": rpc error: code = NotFound desc = could not find container \"9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613\": container with ID starting with 9feccb925c25fc6c90b71d7dd97f5b01c579a917fe1e6b6691aa4455b4428613 not found: ID does not exist"
Nov 02 15:53:56.876863 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.876795    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"]
Nov 02 15:53:56.888862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.888828    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f6f1a0-5479bd9887-frq75"]
Nov 02 15:53:56.936408 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:56.936379    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:57.103602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:57.103514    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:57.292381 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:57.292344    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:57.673671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:57.673374    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:57.863801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:57.863769    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0418a181-53a9-43a2-b261-dac04482277f" path="/var/lib/kubelet/pods/0418a181-53a9-43a2-b261-dac04482277f/volumes"
Nov 02 15:53:57.882420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:57.882385    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:58.179676 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.179623    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"]
Nov 02 15:53:58.180160 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180145    7019 cpu_manager.go:414] "containerMap: removing container" podUID="329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" containerName="revoke-keys"
Nov 02 15:53:58.180207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180162    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" containerName="revoke-keys"
Nov 02 15:53:58.180207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180180    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0418a181-53a9-43a2-b261-dac04482277f" containerName="payload-pre-processing"
Nov 02 15:53:58.180207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180186    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0418a181-53a9-43a2-b261-dac04482277f" containerName="payload-pre-processing"
Nov 02 15:53:58.180207 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180201    7019 cpu_manager.go:414] "containerMap: removing container" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy"
Nov 02 15:53:58.180344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180209    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd4992d-7fc2-4a73-a452-9aa7de71aeaf" containerName="istio-proxy"
Nov 02 15:53:58.180344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180291    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" containerName="revoke-keys"
Nov 02 15:53:58.180344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.180303    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0418a181-53a9-43a2-b261-dac04482277f" containerName="payload-pre-processing"
Nov 02 15:53:58.185038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.185019    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.207558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.207533    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:53:58.207796 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.207777    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:53:58.211690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.210479    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"]
Nov 02 15:53:58.221662 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.218509    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"]
Nov 02 15:53:58.250071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.250039    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.266315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.266281    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"]
Nov 02 15:53:58.350831 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.350791    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.351294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.350841    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.351294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.350979    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdwv\" (UniqueName: \"kubernetes.io/projected/a8eabab3-0500-4f63-9d5f-4b6aa5097970-kube-api-access-tkdwv\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.351294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.351029    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.351294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.351131    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgc2t\" (UniqueName: \"kubernetes.io/projected/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-kube-api-access-lgc2t\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.351294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.351170    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.452742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.452661    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.452742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.452723    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.452974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.452798    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdwv\" (UniqueName: \"kubernetes.io/projected/a8eabab3-0500-4f63-9d5f-4b6aa5097970-kube-api-access-tkdwv\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.452974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.452837    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.452974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.452898    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgc2t\" (UniqueName: \"kubernetes.io/projected/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-kube-api-access-lgc2t\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.452974 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.452934    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.453382 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.453360    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.453766 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.453738    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.455124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.455104    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.455211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.455171    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.478491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.478460    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgc2t\" (UniqueName: \"kubernetes.io/projected/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-kube-api-access-lgc2t\") pod \"maas-api-revoke-sub-aad7088e-nn74x\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.482203 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.482180    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdwv\" (UniqueName: \"kubernetes.io/projected/a8eabab3-0500-4f63-9d5f-4b6aa5097970-kube-api-access-tkdwv\") pod \"maas-api-revoke-sub-b9021546-sd4gd\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.499067 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.499038    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:53:58.589022 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.588949    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:58.602836 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.602809    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:53:58.669317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.668809    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"]
Nov 02 15:53:58.671476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.671448    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-28da58-54g9m"]
Nov 02 15:53:58.747132 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.747048    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:58.810155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.810124    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"]
Nov 02 15:53:58.811546 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:53:58.811513    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8eabab3_0500_4f63_9d5f_4b6aa5097970.slice/crio-7548e71dfadeebeb3ae3fabfe90c8c21a3ecc2c376d421dff64b7143ae6b4be8 WatchSource:0}: Error finding container 7548e71dfadeebeb3ae3fabfe90c8c21a3ecc2c376d421dff64b7143ae6b4be8: Status 404 returned error can't find the container with id 7548e71dfadeebeb3ae3fabfe90c8c21a3ecc2c376d421dff64b7143ae6b4be8
Nov 02 15:53:58.823430 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.823401    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd" event={"ID":"a8eabab3-0500-4f63-9d5f-4b6aa5097970","Type":"ContainerStarted","Data":"7548e71dfadeebeb3ae3fabfe90c8c21a3ecc2c376d421dff64b7143ae6b4be8"}
Nov 02 15:53:58.904629 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:58.904602    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"]
Nov 02 15:53:58.904784 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:53:58.904765    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e50a1ed_55c8_488f_b324_fa12bfe3bb7f.slice/crio-4ff6361035c695cf2699b8c37688db323607e0279de08cd32d7084b3d62c00ce WatchSource:0}: Error finding container 4ff6361035c695cf2699b8c37688db323607e0279de08cd32d7084b3d62c00ce: Status 404 returned error can't find the container with id 4ff6361035c695cf2699b8c37688db323607e0279de08cd32d7084b3d62c00ce
Nov 02 15:53:59.263181 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:59.263132    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:53:59.835350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:59.835308    7019 generic.go:356] "Generic (PLEG): container finished" podID="0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" containerID="13b2d11034ea0fc162fd4b1743b4ecee4489b7d229d76663cdff65e75b3b9ffe" exitCode=0
Nov 02 15:53:59.835853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:59.835399    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x" event={"ID":"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f","Type":"ContainerDied","Data":"13b2d11034ea0fc162fd4b1743b4ecee4489b7d229d76663cdff65e75b3b9ffe"}
Nov 02 15:53:59.835853 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:59.835459    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x" event={"ID":"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f","Type":"ContainerStarted","Data":"4ff6361035c695cf2699b8c37688db323607e0279de08cd32d7084b3d62c00ce"}
Nov 02 15:53:59.836988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:59.836967    7019 generic.go:356] "Generic (PLEG): container finished" podID="a8eabab3-0500-4f63-9d5f-4b6aa5097970" containerID="0566a03b9369f5ff4739530f7e1f3360f254aa3c3d481c88de79971a9da6c932" exitCode=0
Nov 02 15:53:59.837120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:59.837000    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd" event={"ID":"a8eabab3-0500-4f63-9d5f-4b6aa5097970","Type":"ContainerDied","Data":"0566a03b9369f5ff4739530f7e1f3360f254aa3c3d481c88de79971a9da6c932"}
Nov 02 15:53:59.866605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:53:59.862151    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329889be-c09f-4ad4-bcbc-c8fdb63e1dcc" path="/var/lib/kubelet/pods/329889be-c09f-4ad4-bcbc-c8fdb63e1dcc/volumes"
Nov 02 15:54:00.047627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:00.047537    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"]
Nov 02 15:54:00.080703 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:00.080674    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beba5bb9-xh5q9"]
Nov 02 15:54:00.083447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:00.083410    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"]
Nov 02 15:54:00.086400 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:00.086364    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f46db37-vlnpr"]
Nov 02 15:54:00.872671 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:00.871277    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:01.009490 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.009464    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:54:01.047627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.047606    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:54:01.194710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.194675    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8eabab3-0500-4f63-9d5f-4b6aa5097970-kube-api-access-tkdwv\" (UniqueName: \"kubernetes.io/projected/a8eabab3-0500-4f63-9d5f-4b6aa5097970-kube-api-access-tkdwv\") pod \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") "
Nov 02 15:54:01.194951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.194785    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-xks-ca\") pod \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") "
Nov 02 15:54:01.194951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.194855    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-openshift-ca\") pod \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") "
Nov 02 15:54:01.194951 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.194944    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-kube-api-access-lgc2t\" (UniqueName: \"kubernetes.io/projected/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-kube-api-access-lgc2t\") pod \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\" (UID: \"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f\") "
Nov 02 15:54:01.195123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.194999    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-xks-ca\") pod \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") "
Nov 02 15:54:01.195123 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.195030    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-openshift-ca\") pod \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\" (UID: \"a8eabab3-0500-4f63-9d5f-4b6aa5097970\") "
Nov 02 15:54:01.195356 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.195281    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-openshift-ca" pod "0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" (UID: "0e50a1ed-55c8-488f-b324-fa12bfe3bb7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:01.195525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.195478    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.195779 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.195748    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-openshift-ca" pod "a8eabab3-0500-4f63-9d5f-4b6aa5097970" (UID: "a8eabab3-0500-4f63-9d5f-4b6aa5097970"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:01.197556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.197523    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-xks-ca" pod "0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" (UID: "0e50a1ed-55c8-488f-b324-fa12bfe3bb7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:01.197719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.197558    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-kube-api-access-lgc2t" pod "0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" (UID: "0e50a1ed-55c8-488f-b324-fa12bfe3bb7f"). InnerVolumeSpecName "kube-api-access-lgc2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:01.197849 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.197829    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8eabab3-0500-4f63-9d5f-4b6aa5097970-kube-api-access-tkdwv" pod "a8eabab3-0500-4f63-9d5f-4b6aa5097970" (UID: "a8eabab3-0500-4f63-9d5f-4b6aa5097970"). InnerVolumeSpecName "kube-api-access-tkdwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:01.198525 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.198507    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-xks-ca" pod "a8eabab3-0500-4f63-9d5f-4b6aa5097970" (UID: "a8eabab3-0500-4f63-9d5f-4b6aa5097970"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:01.296503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.296466    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.296503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.296499    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgc2t\" (UniqueName: \"kubernetes.io/projected/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f-kube-api-access-lgc2t\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.296503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.296509    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.296776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.296517    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8eabab3-0500-4f63-9d5f-4b6aa5097970-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.296776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.296527    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdwv\" (UniqueName: \"kubernetes.io/projected/a8eabab3-0500-4f63-9d5f-4b6aa5097970-kube-api-access-tkdwv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:01.858856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.858830    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"
Nov 02 15:54:01.860498 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.860471    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"
Nov 02 15:54:01.862298 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.862272    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5375aab-a4a3-4285-b6a6-e0eeb3123a3b" path="/var/lib/kubelet/pods/b5375aab-a4a3-4285-b6a6-e0eeb3123a3b/volumes"
Nov 02 15:54:01.862928 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.862910    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be94e6d6-dfc6-4cdb-9178-55cab00910e6" path="/var/lib/kubelet/pods/be94e6d6-dfc6-4cdb-9178-55cab00910e6/volumes"
Nov 02 15:54:01.863625 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.863603    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x" event={"ID":"0e50a1ed-55c8-488f-b324-fa12bfe3bb7f","Type":"ContainerDied","Data":"4ff6361035c695cf2699b8c37688db323607e0279de08cd32d7084b3d62c00ce"}
Nov 02 15:54:01.863692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.863672    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff6361035c695cf2699b8c37688db323607e0279de08cd32d7084b3d62c00ce"
Nov 02 15:54:01.863692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.863687    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd" event={"ID":"a8eabab3-0500-4f63-9d5f-4b6aa5097970","Type":"ContainerDied","Data":"7548e71dfadeebeb3ae3fabfe90c8c21a3ecc2c376d421dff64b7143ae6b4be8"}
Nov 02 15:54:01.863789 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:01.863714    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7548e71dfadeebeb3ae3fabfe90c8c21a3ecc2c376d421dff64b7143ae6b4be8"
Nov 02 15:54:02.857146 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:02.857117    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:03.970877 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:03.970826    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:54:04.686153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:04.686109    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:06.580572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:06.580533    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:08.228333 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:08.228282    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:08.424367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:08.424332    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:08.550954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:08.550859    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"]
Nov 02 15:54:08.551149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:08.551071    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy" containerID="cri-o://9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477" gracePeriod=30
Nov 02 15:54:08.630338 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:08.630294    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:09.063055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.063013    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:09.229990 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.229956    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:09.330712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.330620    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"]
Nov 02 15:54:09.331143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.331126    7019 cpu_manager.go:414] "containerMap: removing container" podUID="0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" containerName="revoke-keys"
Nov 02 15:54:09.331188 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.331145    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" containerName="revoke-keys"
Nov 02 15:54:09.331238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.331194    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a8eabab3-0500-4f63-9d5f-4b6aa5097970" containerName="revoke-keys"
Nov 02 15:54:09.331238 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.331199    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8eabab3-0500-4f63-9d5f-4b6aa5097970" containerName="revoke-keys"
Nov 02 15:54:09.331311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.331275    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8eabab3-0500-4f63-9d5f-4b6aa5097970" containerName="revoke-keys"
Nov 02 15:54:09.331311 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.331287    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" containerName="revoke-keys"
Nov 02 15:54:09.336015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.335997    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.355979 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.355946    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:54:09.358134 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.358111    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"]
Nov 02 15:54:09.367941 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.367918    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:54:09.422410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.422371    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.422596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.422419    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.422680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.422622    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmnp2\" (UniqueName: \"kubernetes.io/projected/3b14967b-0e2c-44da-a9a7-853d082ead96-kube-api-access-tmnp2\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.523852 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.523809    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmnp2\" (UniqueName: \"kubernetes.io/projected/3b14967b-0e2c-44da-a9a7-853d082ead96-kube-api-access-tmnp2\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.524069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.523929    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.524069 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.523966    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.524637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.524611    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.526457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.526432    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.552126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.552090    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmnp2\" (UniqueName: \"kubernetes.io/projected/3b14967b-0e2c-44da-a9a7-853d082ead96-kube-api-access-tmnp2\") pod \"maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.650108 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.650077    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:09.688440 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.688397    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:09.965718 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:09.965686    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"]
Nov 02 15:54:09.966086 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:54:09.966047    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b14967b_0e2c_44da_a9a7_853d082ead96.slice/crio-46c334edd865f23f18de6cb247a66a29537e1c25f80c558872a7ad7bc47c1df8 WatchSource:0}: Error finding container 46c334edd865f23f18de6cb247a66a29537e1c25f80c558872a7ad7bc47c1df8: Status 404 returned error can't find the container with id 46c334edd865f23f18de6cb247a66a29537e1c25f80c558872a7ad7bc47c1df8
Nov 02 15:54:10.600802 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.600688    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"]
Nov 02 15:54:10.601341 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.600920    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb" podUID="585b1556-e70a-4d65-905e-9d0bb0555eb2" containerName="maas-api" containerID="cri-o://736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8" gracePeriod=30
Nov 02 15:54:10.736010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.735617    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:10.932256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.930544    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:54:10.936357 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.936329    7019 generic.go:356] "Generic (PLEG): container finished" podID="3b14967b-0e2c-44da-a9a7-853d082ead96" containerID="4e075bbc389f69455bc9a13f28113d13161c0a562046e43ecb5a78bfc9e4e8b4" exitCode=0
Nov 02 15:54:10.936485 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.936414    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk" event={"ID":"3b14967b-0e2c-44da-a9a7-853d082ead96","Type":"ContainerDied","Data":"4e075bbc389f69455bc9a13f28113d13161c0a562046e43ecb5a78bfc9e4e8b4"}
Nov 02 15:54:10.936552 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.936483    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk" event={"ID":"3b14967b-0e2c-44da-a9a7-853d082ead96","Type":"ContainerStarted","Data":"46c334edd865f23f18de6cb247a66a29537e1c25f80c558872a7ad7bc47c1df8"}
Nov 02 15:54:10.937825 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.937803    7019 generic.go:356] "Generic (PLEG): container finished" podID="585b1556-e70a-4d65-905e-9d0bb0555eb2" containerID="736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8" exitCode=0
Nov 02 15:54:10.937961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.937841    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"
Nov 02 15:54:10.937961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.937883    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb" event={"ID":"585b1556-e70a-4d65-905e-9d0bb0555eb2","Type":"ContainerDied","Data":"736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8"}
Nov 02 15:54:10.937961 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.937932    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb" event={"ID":"585b1556-e70a-4d65-905e-9d0bb0555eb2","Type":"ContainerDied","Data":"15482fc6805649be1c92cb3832849ed2f3ae57d1f6654465dbc13ecd0656c310"}
Nov 02 15:54:10.938143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.937962    7019 scope.go:122] "RemoveContainer" containerID="736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8"
Nov 02 15:54:10.950116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.950092    7019 scope.go:122] "RemoveContainer" containerID="736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8"
Nov 02 15:54:10.950422 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:10.950398    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8\": container with ID starting with 736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8 not found: ID does not exist" containerID="736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8"
Nov 02 15:54:10.950505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:10.950430    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8"} err="failed to get container status \"736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8\": rpc error: code = NotFound desc = could not find container \"736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8\": container with ID starting with 736f875d651793a81a6df3ccab74dcfa984fe627eb0be4febe59ff9251e424c8 not found: ID does not exist"
Nov 02 15:54:11.043337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.043289    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585b1556-e70a-4d65-905e-9d0bb0555eb2-kube-api-access-htgpc\" (UniqueName: \"kubernetes.io/projected/585b1556-e70a-4d65-905e-9d0bb0555eb2-kube-api-access-htgpc\") pod \"585b1556-e70a-4d65-905e-9d0bb0555eb2\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") "
Nov 02 15:54:11.043337 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.043326    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585b1556-e70a-4d65-905e-9d0bb0555eb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585b1556-e70a-4d65-905e-9d0bb0555eb2-openshift-service-ca\") pod \"585b1556-e70a-4d65-905e-9d0bb0555eb2\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") "
Nov 02 15:54:11.043619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.043357    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-metrics-certs\") pod \"585b1556-e70a-4d65-905e-9d0bb0555eb2\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") "
Nov 02 15:54:11.043619 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.043516    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-maas-api-tls\") pod \"585b1556-e70a-4d65-905e-9d0bb0555eb2\" (UID: \"585b1556-e70a-4d65-905e-9d0bb0555eb2\") "
Nov 02 15:54:11.044083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.043852    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585b1556-e70a-4d65-905e-9d0bb0555eb2-openshift-service-ca" pod "585b1556-e70a-4d65-905e-9d0bb0555eb2" (UID: "585b1556-e70a-4d65-905e-9d0bb0555eb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:11.044083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.044017    7019 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/585b1556-e70a-4d65-905e-9d0bb0555eb2-openshift-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:11.045686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.045632    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-metrics-certs" pod "585b1556-e70a-4d65-905e-9d0bb0555eb2" (UID: "585b1556-e70a-4d65-905e-9d0bb0555eb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:11.045754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.045687    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585b1556-e70a-4d65-905e-9d0bb0555eb2-kube-api-access-htgpc" pod "585b1556-e70a-4d65-905e-9d0bb0555eb2" (UID: "585b1556-e70a-4d65-905e-9d0bb0555eb2"). InnerVolumeSpecName "kube-api-access-htgpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:11.045754 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.045697    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-maas-api-tls" pod "585b1556-e70a-4d65-905e-9d0bb0555eb2" (UID: "585b1556-e70a-4d65-905e-9d0bb0555eb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:11.145153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.145054    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-maas-api-tls\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:11.145153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.145088    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htgpc\" (UniqueName: \"kubernetes.io/projected/585b1556-e70a-4d65-905e-9d0bb0555eb2-kube-api-access-htgpc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:11.145153 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.145098    7019 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/585b1556-e70a-4d65-905e-9d0bb0555eb2-metrics-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:11.293529 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.293495    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"]
Nov 02 15:54:11.299860 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.299833    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-739bf1-78cd79df6c-zcpqb"]
Nov 02 15:54:11.337457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.337419    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:11.428627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.428542    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"]
Nov 02 15:54:11.430658 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.430619    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-739bf1-vvz6q"]
Nov 02 15:54:11.516720 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.516687    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:11.861256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.861218    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c87b12-0c06-4d00-996a-59978391e30f" path="/var/lib/kubelet/pods/33c87b12-0c06-4d00-996a-59978391e30f/volumes"
Nov 02 15:54:11.861848 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.861828    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585b1556-e70a-4d65-905e-9d0bb0555eb2" path="/var/lib/kubelet/pods/585b1556-e70a-4d65-905e-9d0bb0555eb2/volumes"
Nov 02 15:54:11.929978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:11.929938    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:12.013981 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.013941    7019 patch_prober.go:29] interesting pod/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:54:12.014138 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.014007    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:54:12.097883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.097856    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:12.155000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.154918    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-openshift-ca\") pod \"3b14967b-0e2c-44da-a9a7-853d082ead96\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") "
Nov 02 15:54:12.155179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.155016    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b14967b-0e2c-44da-a9a7-853d082ead96-kube-api-access-tmnp2\" (UniqueName: \"kubernetes.io/projected/3b14967b-0e2c-44da-a9a7-853d082ead96-kube-api-access-tmnp2\") pod \"3b14967b-0e2c-44da-a9a7-853d082ead96\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") "
Nov 02 15:54:12.155179 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.155079    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-xks-ca\") pod \"3b14967b-0e2c-44da-a9a7-853d082ead96\" (UID: \"3b14967b-0e2c-44da-a9a7-853d082ead96\") "
Nov 02 15:54:12.155377 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.155259    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-openshift-ca" pod "3b14967b-0e2c-44da-a9a7-853d082ead96" (UID: "3b14967b-0e2c-44da-a9a7-853d082ead96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:12.155590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.155560    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:12.157193 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.157168    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-xks-ca" pod "3b14967b-0e2c-44da-a9a7-853d082ead96" (UID: "3b14967b-0e2c-44da-a9a7-853d082ead96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:12.157317 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.157254    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b14967b-0e2c-44da-a9a7-853d082ead96-kube-api-access-tmnp2" pod "3b14967b-0e2c-44da-a9a7-853d082ead96" (UID: "3b14967b-0e2c-44da-a9a7-853d082ead96"). InnerVolumeSpecName "kube-api-access-tmnp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:12.256415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.256377    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmnp2\" (UniqueName: \"kubernetes.io/projected/3b14967b-0e2c-44da-a9a7-853d082ead96-kube-api-access-tmnp2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:12.256415 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.256409    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b14967b-0e2c-44da-a9a7-853d082ead96-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:12.953591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.953553    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"
Nov 02 15:54:12.954031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.953591    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk" event={"ID":"3b14967b-0e2c-44da-a9a7-853d082ead96","Type":"ContainerDied","Data":"46c334edd865f23f18de6cb247a66a29537e1c25f80c558872a7ad7bc47c1df8"}
Nov 02 15:54:12.954031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:12.953640    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46c334edd865f23f18de6cb247a66a29537e1c25f80c558872a7ad7bc47c1df8"
Nov 02 15:54:13.300294 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.300194    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:13.863198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.863172    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:54:13.962737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.962705    7019 generic.go:356] "Generic (PLEG): container finished" podID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerID="9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477" exitCode=0
Nov 02 15:54:13.963161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.962766    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"
Nov 02 15:54:13.963161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.962792    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" event={"ID":"ad1943de-bc74-4783-aeef-552f906a8f7a","Type":"ContainerDied","Data":"9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477"}
Nov 02 15:54:13.963161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.962855    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5" event={"ID":"ad1943de-bc74-4783-aeef-552f906a8f7a","Type":"ContainerDied","Data":"93fde9b33f4cde7e127534d9ac635a6783770dbf56ceee8fbf5e594a45e866b7"}
Nov 02 15:54:13.963161 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.962886    7019 scope.go:122] "RemoveContainer" containerID="9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477"
Nov 02 15:54:13.973002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.972982    7019 scope.go:122] "RemoveContainer" containerID="4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b"
Nov 02 15:54:13.976335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976312    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1943de-bc74-4783-aeef-552f906a8f7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1943de-bc74-4783-aeef-552f906a8f7a-istiod-ca-cert\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976385    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-kube-api-access-j75fn\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-kube-api-access-j75fn\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976465 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976414    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-socket\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976512    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-data\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976540    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-credential-socket\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976559    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-certs\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976602    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-envoy\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976667    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-podinfo\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.976827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.976693    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-token\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-token\") pod \"ad1943de-bc74-4783-aeef-552f906a8f7a\" (UID: \"ad1943de-bc74-4783-aeef-552f906a8f7a\") "
Nov 02 15:54:13.977743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.977250    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-data" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:13.977743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.977341    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-certs" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:13.977743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.977382    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-credential-socket" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:13.977743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.977597    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-socket" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:13.978044 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.977754    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1943de-bc74-4783-aeef-552f906a8f7a-istiod-ca-cert" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:13.979769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.979689    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-kube-api-access-j75fn" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "kube-api-access-j75fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:13.979769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.979748    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-token" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:13.980774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.980742    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-envoy" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:13.980859 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.980748    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-podinfo" pod "ad1943de-bc74-4783-aeef-552f906a8f7a" (UID: "ad1943de-bc74-4783-aeef-552f906a8f7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:54:13.984574 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.984534    7019 scope.go:122] "RemoveContainer" containerID="9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477"
Nov 02 15:54:13.984887 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:13.984868    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477\": container with ID starting with 9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477 not found: ID does not exist" containerID="9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477"
Nov 02 15:54:13.984984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.984899    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477"} err="failed to get container status \"9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477\": rpc error: code = NotFound desc = could not find container \"9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477\": container with ID starting with 9217dfac8c488458f57a423e09767a747df885edc7753f251a6b6e8d1570f477 not found: ID does not exist"
Nov 02 15:54:13.984984 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.984927    7019 scope.go:122] "RemoveContainer" containerID="4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b"
Nov 02 15:54:13.985241 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:13.985208    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b\": container with ID starting with 4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b not found: ID does not exist" containerID="4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b"
Nov 02 15:54:13.985350 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:13.985244    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b"} err="failed to get container status \"4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b\": rpc error: code = NotFound desc = could not find container \"4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b\": container with ID starting with 4f2ca5b8c6c00bb0d56828794da765d649c0a56bf5d07d277087111b1a98a46b not found: ID does not exist"
Nov 02 15:54:14.078427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078382    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1943de-bc74-4783-aeef-552f906a8f7a-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078427 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078428    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j75fn\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-kube-api-access-j75fn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078445    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078459    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078472    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078485    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078498    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078510    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.078744 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.078521    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1943de-bc74-4783-aeef-552f906a8f7a-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:14.267239 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.267195    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:14.341062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.341027    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"]
Nov 02 15:54:14.352256 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.352207    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-28da58-openshift-default-76446c9f79-7f2j5"]
Nov 02 15:54:14.794697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:14.794633    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:15.138002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:15.137962    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:15.861088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:15.861052    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" path="/var/lib/kubelet/pods/ad1943de-bc74-4783-aeef-552f906a8f7a/volumes"
Nov 02 15:54:16.146539 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:16.146451    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:16.534968 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:16.534848    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:16.728308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:16.728266    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:17.039656 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:17.039599    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:17.238693 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:17.238628    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:18.733923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:18.733874    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:18.961476 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:18.961443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:19.139952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:19.139920    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:19.422611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:19.422497    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:19.627591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:19.627557    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:19.810770 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:19.810686    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:20.109085 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:20.108989    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:20.341778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:20.341738    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:20.581107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:20.581070    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:21.824767 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:21.824715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:22.200784 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:22.200747    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:22.514437 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:22.514340    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:22.584727 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:22.584691    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"]
Nov 02 15:54:22.584939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:22.584898    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy" containerID="cri-o://6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad" gracePeriod=30
Nov 02 15:54:22.708313 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:22.708265    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:54:22.829939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:22.829839    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:23.591110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591068    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"]
Nov 02 15:54:23.591542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591527    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy"
Nov 02 15:54:23.591542 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591541    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy"
Nov 02 15:54:23.591700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591573    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3b14967b-0e2c-44da-a9a7-853d082ead96" containerName="revoke-keys"
Nov 02 15:54:23.591700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591578    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b14967b-0e2c-44da-a9a7-853d082ead96" containerName="revoke-keys"
Nov 02 15:54:23.591700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591602    7019 cpu_manager.go:414] "containerMap: removing container" podUID="585b1556-e70a-4d65-905e-9d0bb0555eb2" containerName="maas-api"
Nov 02 15:54:23.591700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591610    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585b1556-e70a-4d65-905e-9d0bb0555eb2" containerName="maas-api"
Nov 02 15:54:23.591700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591681    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy"
Nov 02 15:54:23.591700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591689    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="585b1556-e70a-4d65-905e-9d0bb0555eb2" containerName="maas-api"
Nov 02 15:54:23.591700 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591698    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy"
Nov 02 15:54:23.592072 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.591708    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b14967b-0e2c-44da-a9a7-853d082ead96" containerName="revoke-keys"
Nov 02 15:54:23.594776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.594754    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.606722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.600300    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"]
Nov 02 15:54:23.606722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.603760    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy"
Nov 02 15:54:23.606722 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.603780    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1943de-bc74-4783-aeef-552f906a8f7a" containerName="istio-proxy"
Nov 02 15:54:23.611470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.611437    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.615730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.615702    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:54:23.615866 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.615797    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:54:23.635462 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.635430    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"]
Nov 02 15:54:23.637697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.637670    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"]
Nov 02 15:54:23.715201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.715163    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.715201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.715206    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5bq\" (UniqueName: \"kubernetes.io/projected/bdf9cf03-3a92-418d-a9b9-72caba0ad432-kube-api-access-nm5bq\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.715425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.715275    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.715425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.715332    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.715514 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.715485    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thn2\" (UniqueName: \"kubernetes.io/projected/908b243c-48a9-4485-a338-6cc361f65ad8-kube-api-access-5thn2\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.715604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.715587    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.816330 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.816295    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thn2\" (UniqueName: \"kubernetes.io/projected/908b243c-48a9-4485-a338-6cc361f65ad8-kube-api-access-5thn2\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.816518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.816346    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.816518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.816395    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.816518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.816416    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5bq\" (UniqueName: \"kubernetes.io/projected/bdf9cf03-3a92-418d-a9b9-72caba0ad432-kube-api-access-nm5bq\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.816518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.816437    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.816518 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.816494    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.817171 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.817139    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.817325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.817239    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.818905 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.818877    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.819033 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.819016    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.845769 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.845701    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thn2\" (UniqueName: \"kubernetes.io/projected/908b243c-48a9-4485-a338-6cc361f65ad8-kube-api-access-5thn2\") pod \"maas-api-revoke-sub-a8b5d512-p6s6v\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.846868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.846847    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5bq\" (UniqueName: \"kubernetes.io/projected/bdf9cf03-3a92-418d-a9b9-72caba0ad432-kube-api-access-nm5bq\") pod \"maas-api-revoke-sub-0223a1d9-xwd5r\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:23.915410 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.915376    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:23.930782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:23.930755    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:24.010569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:24.010388    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:24.245886 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:54:24.245855    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908b243c_48a9_4485_a338_6cc361f65ad8.slice/crio-a43b5d6883cf66e8329ef7a03db55ca7b4000f78a0d79530086f7fc660bbaf2c WatchSource:0}: Error finding container a43b5d6883cf66e8329ef7a03db55ca7b4000f78a0d79530086f7fc660bbaf2c: Status 404 returned error can't find the container with id a43b5d6883cf66e8329ef7a03db55ca7b4000f78a0d79530086f7fc660bbaf2c
Nov 02 15:54:24.246736 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:24.246712    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"]
Nov 02 15:54:24.274017 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:24.273992    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"]
Nov 02 15:54:24.274528 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:54:24.274497    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf9cf03_3a92_418d_a9b9_72caba0ad432.slice/crio-766ee2be45fd3b6e65ee4f74d4ed3bf625777b96c525988c751de3768d6f3999 WatchSource:0}: Error finding container 766ee2be45fd3b6e65ee4f74d4ed3bf625777b96c525988c751de3768d6f3999: Status 404 returned error can't find the container with id 766ee2be45fd3b6e65ee4f74d4ed3bf625777b96c525988c751de3768d6f3999
Nov 02 15:54:24.485563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:24.485481    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:54:25.051002 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:25.050963    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v" event={"ID":"908b243c-48a9-4485-a338-6cc361f65ad8","Type":"ContainerStarted","Data":"d7548483b7bf5926523e00dc740aa8dc6d690a283db2b23908bd74688ba67308"}
Nov 02 15:54:25.051471 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:25.051029    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v" event={"ID":"908b243c-48a9-4485-a338-6cc361f65ad8","Type":"ContainerStarted","Data":"a43b5d6883cf66e8329ef7a03db55ca7b4000f78a0d79530086f7fc660bbaf2c"}
Nov 02 15:54:25.052757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:25.052729    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r" event={"ID":"bdf9cf03-3a92-418d-a9b9-72caba0ad432","Type":"ContainerStarted","Data":"9e135ce84146743d906eda7757e91ba4524cdca3871e802f2f500acca3c8ce4c"}
Nov 02 15:54:25.052876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:25.052777    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r" event={"ID":"bdf9cf03-3a92-418d-a9b9-72caba0ad432","Type":"ContainerStarted","Data":"766ee2be45fd3b6e65ee4f74d4ed3bf625777b96c525988c751de3768d6f3999"}
Nov 02 15:54:25.171205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:25.171141    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v" podStartSLOduration=2.171125124 podStartE2EDuration="2.171125124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:54:25.120393095 +0000 UTC m=+3459.374346720" watchObservedRunningTime="2026-11-02 15:54:25.171125124 +0000 UTC m=+3459.425078749"
Nov 02 15:54:25.173068 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:25.172990    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r" podStartSLOduration=2.172981765 podStartE2EDuration="2.172981765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:54:25.170843675 +0000 UTC m=+3459.424797298" watchObservedRunningTime="2026-11-02 15:54:25.172981765 +0000 UTC m=+3459.426935389"
Nov 02 15:54:25.807855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:25.807817    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:26.060816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:26.060728    7019 generic.go:356] "Generic (PLEG): container finished" podID="908b243c-48a9-4485-a338-6cc361f65ad8" containerID="d7548483b7bf5926523e00dc740aa8dc6d690a283db2b23908bd74688ba67308" exitCode=0
Nov 02 15:54:26.061255 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:26.060822    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v" event={"ID":"908b243c-48a9-4485-a338-6cc361f65ad8","Type":"ContainerDied","Data":"d7548483b7bf5926523e00dc740aa8dc6d690a283db2b23908bd74688ba67308"}
Nov 02 15:54:26.062602 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:26.062579    7019 generic.go:356] "Generic (PLEG): container finished" podID="bdf9cf03-3a92-418d-a9b9-72caba0ad432" containerID="9e135ce84146743d906eda7757e91ba4524cdca3871e802f2f500acca3c8ce4c" exitCode=0
Nov 02 15:54:26.062733 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:26.062675    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r" event={"ID":"bdf9cf03-3a92-418d-a9b9-72caba0ad432","Type":"ContainerDied","Data":"9e135ce84146743d906eda7757e91ba4524cdca3871e802f2f500acca3c8ce4c"}
Nov 02 15:54:27.231572 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.231546    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:27.258197 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.258172    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:27.359220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359130    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-openshift-ca\") pod \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") "
Nov 02 15:54:27.359220 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359173    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-xks-ca\") pod \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") "
Nov 02 15:54:27.359474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359403    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-openshift-ca\") pod \"908b243c-48a9-4485-a338-6cc361f65ad8\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") "
Nov 02 15:54:27.359474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359464    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908b243c-48a9-4485-a338-6cc361f65ad8-kube-api-access-5thn2\" (UniqueName: \"kubernetes.io/projected/908b243c-48a9-4485-a338-6cc361f65ad8-kube-api-access-5thn2\") pod \"908b243c-48a9-4485-a338-6cc361f65ad8\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") "
Nov 02 15:54:27.359595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359505    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-xks-ca\") pod \"908b243c-48a9-4485-a338-6cc361f65ad8\" (UID: \"908b243c-48a9-4485-a338-6cc361f65ad8\") "
Nov 02 15:54:27.359595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359542    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf9cf03-3a92-418d-a9b9-72caba0ad432-kube-api-access-nm5bq\" (UniqueName: \"kubernetes.io/projected/bdf9cf03-3a92-418d-a9b9-72caba0ad432-kube-api-access-nm5bq\") pod \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\" (UID: \"bdf9cf03-3a92-418d-a9b9-72caba0ad432\") "
Nov 02 15:54:27.359595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359538    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-openshift-ca" pod "bdf9cf03-3a92-418d-a9b9-72caba0ad432" (UID: "bdf9cf03-3a92-418d-a9b9-72caba0ad432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:27.360060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.359770    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-openshift-ca" pod "908b243c-48a9-4485-a338-6cc361f65ad8" (UID: "908b243c-48a9-4485-a338-6cc361f65ad8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:27.360060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.360036    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.360060 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.360057    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.361931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.361905    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-xks-ca" pod "908b243c-48a9-4485-a338-6cc361f65ad8" (UID: "908b243c-48a9-4485-a338-6cc361f65ad8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:27.362041 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.361994    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf9cf03-3a92-418d-a9b9-72caba0ad432-kube-api-access-nm5bq" pod "bdf9cf03-3a92-418d-a9b9-72caba0ad432" (UID: "bdf9cf03-3a92-418d-a9b9-72caba0ad432"). InnerVolumeSpecName "kube-api-access-nm5bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:27.362124 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.362077    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908b243c-48a9-4485-a338-6cc361f65ad8-kube-api-access-5thn2" pod "908b243c-48a9-4485-a338-6cc361f65ad8" (UID: "908b243c-48a9-4485-a338-6cc361f65ad8"). InnerVolumeSpecName "kube-api-access-5thn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:27.362166 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.362122    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-xks-ca" pod "bdf9cf03-3a92-418d-a9b9-72caba0ad432" (UID: "bdf9cf03-3a92-418d-a9b9-72caba0ad432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:27.461021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.460982    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thn2\" (UniqueName: \"kubernetes.io/projected/908b243c-48a9-4485-a338-6cc361f65ad8-kube-api-access-5thn2\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.461021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.461014    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908b243c-48a9-4485-a338-6cc361f65ad8-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.461021 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.461027    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm5bq\" (UniqueName: \"kubernetes.io/projected/bdf9cf03-3a92-418d-a9b9-72caba0ad432-kube-api-access-nm5bq\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.461273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.461039    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdf9cf03-3a92-418d-a9b9-72caba0ad432-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.819631 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.819599    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:27.829504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.829477    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:54:27.965963 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.965927    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-kube-api-access-9knvs\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-kube-api-access-9knvs\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.965974    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-istio-token\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-istio-token\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966006    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc91941e-07ef-4265-8a68-78be32735b89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc91941e-07ef-4265-8a68-78be32735b89-istiod-ca-cert\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966032    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-data\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966058    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-certs\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966097    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-envoy\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966197    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc91941e-07ef-4265-8a68-78be32735b89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc91941e-07ef-4265-8a68-78be32735b89-istio-podinfo\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966290    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-credential-socket\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966319    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-socket\") pod \"fc91941e-07ef-4265-8a68-78be32735b89\" (UID: \"fc91941e-07ef-4265-8a68-78be32735b89\") "
Nov 02 15:54:27.966457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966365    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-certs" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:27.966457 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966381    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc91941e-07ef-4265-8a68-78be32735b89-istiod-ca-cert" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:27.966763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966582    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-credential-socket" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:27.966763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966723    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-data" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:27.966954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966888    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.966954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966925    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc91941e-07ef-4265-8a68-78be32735b89-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.966954 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966936    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-socket" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:27.967151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966942    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.967151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.966981    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:27.968506 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.968472    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-envoy" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:27.968690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.968663    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-istio-token" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:27.968776 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.968743    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-kube-api-access-9knvs" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "kube-api-access-9knvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:27.968895 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:27.968878    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc91941e-07ef-4265-8a68-78be32735b89-istio-podinfo" pod "fc91941e-07ef-4265-8a68-78be32735b89" (UID: "fc91941e-07ef-4265-8a68-78be32735b89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:54:28.068081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.068044    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:28.068081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.068074    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc91941e-07ef-4265-8a68-78be32735b89-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:28.068081 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.068083    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc91941e-07ef-4265-8a68-78be32735b89-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:28.068325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.068092    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9knvs\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-kube-api-access-9knvs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:28.068325 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.068102    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc91941e-07ef-4265-8a68-78be32735b89-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:28.079024 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.078987    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r" event={"ID":"bdf9cf03-3a92-418d-a9b9-72caba0ad432","Type":"ContainerDied","Data":"766ee2be45fd3b6e65ee4f74d4ed3bf625777b96c525988c751de3768d6f3999"}
Nov 02 15:54:28.079190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.079074    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="766ee2be45fd3b6e65ee4f74d4ed3bf625777b96c525988c751de3768d6f3999"
Nov 02 15:54:28.079190 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.079003    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"
Nov 02 15:54:28.080472 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.080452    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"
Nov 02 15:54:28.080589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.080499    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v" event={"ID":"908b243c-48a9-4485-a338-6cc361f65ad8","Type":"ContainerDied","Data":"a43b5d6883cf66e8329ef7a03db55ca7b4000f78a0d79530086f7fc660bbaf2c"}
Nov 02 15:54:28.080589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.080543    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a43b5d6883cf66e8329ef7a03db55ca7b4000f78a0d79530086f7fc660bbaf2c"
Nov 02 15:54:28.082224 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.082197    7019 generic.go:356] "Generic (PLEG): container finished" podID="fc91941e-07ef-4265-8a68-78be32735b89" containerID="6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad" exitCode=0
Nov 02 15:54:28.082346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.082248    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"
Nov 02 15:54:28.082346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.082249    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" event={"ID":"fc91941e-07ef-4265-8a68-78be32735b89","Type":"ContainerDied","Data":"6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad"}
Nov 02 15:54:28.082479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.082375    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph" event={"ID":"fc91941e-07ef-4265-8a68-78be32735b89","Type":"ContainerDied","Data":"f298177d1970d56510d6fb4da441246ed7a8b739dc6078d59dc28083a3244adb"}
Nov 02 15:54:28.082479 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.082411    7019 scope.go:122] "RemoveContainer" containerID="6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad"
Nov 02 15:54:28.096037 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.096009    7019 scope.go:122] "RemoveContainer" containerID="6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad"
Nov 02 15:54:28.096470 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:28.096439    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad\": container with ID starting with 6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad not found: ID does not exist" containerID="6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad"
Nov 02 15:54:28.096541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.096477    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad"} err="failed to get container status \"6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad\": rpc error: code = NotFound desc = could not find container \"6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad\": container with ID starting with 6b14007fcd73443af9cfe373731be63b42049350d60097bba28f23b4eaadc0ad not found: ID does not exist"
Nov 02 15:54:28.156309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.156229    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"]
Nov 02 15:54:28.160469 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.160444    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-739bf1-openshift-default-7585fd4fbf-jvwph"]
Nov 02 15:54:28.405580 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.405535    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:28.642511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.642470    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:28.717752 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.717718    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"]
Nov 02 15:54:28.718039 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.718009    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy" containerID="cri-o://b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d" gracePeriod=30
Nov 02 15:54:28.862020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:28.861985    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:29.379151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379117    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"]
Nov 02 15:54:29.379639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379621    7019 cpu_manager.go:414] "containerMap: removing container" podUID="bdf9cf03-3a92-418d-a9b9-72caba0ad432" containerName="revoke-keys"
Nov 02 15:54:29.379639 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379639    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf9cf03-3a92-418d-a9b9-72caba0ad432" containerName="revoke-keys"
Nov 02 15:54:29.379792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379693    7019 cpu_manager.go:414] "containerMap: removing container" podUID="908b243c-48a9-4485-a338-6cc361f65ad8" containerName="revoke-keys"
Nov 02 15:54:29.379792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379700    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908b243c-48a9-4485-a338-6cc361f65ad8" containerName="revoke-keys"
Nov 02 15:54:29.379792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379714    7019 cpu_manager.go:414] "containerMap: removing container" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy"
Nov 02 15:54:29.379792 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379719    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy"
Nov 02 15:54:29.379969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379799    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="908b243c-48a9-4485-a338-6cc361f65ad8" containerName="revoke-keys"
Nov 02 15:54:29.379969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379811    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc91941e-07ef-4265-8a68-78be32735b89" containerName="istio-proxy"
Nov 02 15:54:29.379969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.379821    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf9cf03-3a92-418d-a9b9-72caba0ad432" containerName="revoke-keys"
Nov 02 15:54:29.384737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.384715    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.409504 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.409474    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"]
Nov 02 15:54:29.483969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.483932    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zsxl\" (UniqueName: \"kubernetes.io/projected/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-kube-api-access-9zsxl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.484156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.484054    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.484156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.484083    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.487823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.487467    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:54:29.487964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.487907    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:54:29.585162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.585127    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.585162 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.585170    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.585385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.585238    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zsxl\" (UniqueName: \"kubernetes.io/projected/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-kube-api-access-9zsxl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.585830 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.585810    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.587520 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.587460    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.612133 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.612104    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zsxl\" (UniqueName: \"kubernetes.io/projected/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-kube-api-access-9zsxl\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.690763 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.690642    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:29.698464 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.698436    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:29.860482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:29.860447    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc91941e-07ef-4265-8a68-78be32735b89" path="/var/lib/kubelet/pods/fc91941e-07ef-4265-8a68-78be32735b89/volumes"
Nov 02 15:54:30.018694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:30.018669    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"]
Nov 02 15:54:30.022074 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:54:30.022045    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3fec70_15ac_48fa_ac7d_ff7da5997d6c.slice/crio-37e767830e5827e3f2b223a436df91fe9bc6c663d74138e81818747bd76105ed WatchSource:0}: Error finding container 37e767830e5827e3f2b223a436df91fe9bc6c663d74138e81818747bd76105ed: Status 404 returned error can't find the container with id 37e767830e5827e3f2b223a436df91fe9bc6c663d74138e81818747bd76105ed
Nov 02 15:54:30.097447 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:30.097413    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x" event={"ID":"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c","Type":"ContainerStarted","Data":"37e767830e5827e3f2b223a436df91fe9bc6c663d74138e81818747bd76105ed"}
Nov 02 15:54:30.128927 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:30.128894    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:31.106487 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:31.106452    7019 generic.go:356] "Generic (PLEG): container finished" podID="6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" containerID="59058f7075d00028bb6e1f51037a132c03007423d592f13594c64fea051e8294" exitCode=0
Nov 02 15:54:31.106945 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:31.106537    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x" event={"ID":"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c","Type":"ContainerDied","Data":"59058f7075d00028bb6e1f51037a132c03007423d592f13594c64fea051e8294"}
Nov 02 15:54:31.517083 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:31.517036    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:32.259164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.259138    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:32.316164 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.316133    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-kube-api-access-9zsxl\" (UniqueName: \"kubernetes.io/projected/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-kube-api-access-9zsxl\") pod \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") "
Nov 02 15:54:32.316346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.316234    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-xks-ca\") pod \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") "
Nov 02 15:54:32.316346 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.316266    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-openshift-ca\") pod \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\" (UID: \"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c\") "
Nov 02 15:54:32.316780 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.316751    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-openshift-ca" pod "6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" (UID: "6d3fec70-15ac-48fa-ac7d-ff7da5997d6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:32.318388 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.318363    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-xks-ca" pod "6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" (UID: "6d3fec70-15ac-48fa-ac7d-ff7da5997d6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:32.318973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.318957    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-kube-api-access-9zsxl" pod "6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" (UID: "6d3fec70-15ac-48fa-ac7d-ff7da5997d6c"). InnerVolumeSpecName "kube-api-access-9zsxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:32.418111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.418031    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zsxl\" (UniqueName: \"kubernetes.io/projected/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-kube-api-access-9zsxl\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:32.418111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.418062    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:32.418111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:32.418073    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:33.122339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:33.122305    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x" event={"ID":"6d3fec70-15ac-48fa-ac7d-ff7da5997d6c","Type":"ContainerDied","Data":"37e767830e5827e3f2b223a436df91fe9bc6c663d74138e81818747bd76105ed"}
Nov 02 15:54:33.122339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:33.122336    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"
Nov 02 15:54:33.122543 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:33.122365    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37e767830e5827e3f2b223a436df91fe9bc6c663d74138e81818747bd76105ed"
Nov 02 15:54:33.126719 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:33.126690    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:33.987989 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:33.987929    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:54:34.036063 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036026    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-istio-token\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-istio-token\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036076    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-envoy\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036193    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e77765a7-6626-4005-b10c-db1068f65c21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77765a7-6626-4005-b10c-db1068f65c21-istio-podinfo\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036227    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-credential-socket\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036279    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-data\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036321    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-socket\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036420 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036361    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-certs\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036462    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-kube-api-access-lffc8\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-kube-api-access-lffc8\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036491    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77765a7-6626-4005-b10c-db1068f65c21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77765a7-6626-4005-b10c-db1068f65c21-istiod-ca-cert\") pod \"e77765a7-6626-4005-b10c-db1068f65c21\" (UID: \"e77765a7-6626-4005-b10c-db1068f65c21\") "
Nov 02 15:54:34.036596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036516    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-credential-socket" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:34.036596 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036540    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-data" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:34.036856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036673    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-socket" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:34.036856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036742    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-certs" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:34.036964 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036945    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.037020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036964    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.037020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036979    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.037020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036994    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.037020 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.036992    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77765a7-6626-4005-b10c-db1068f65c21-istiod-ca-cert" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:34.038474 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.038451    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-istio-token" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:34.038588 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.038494    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e77765a7-6626-4005-b10c-db1068f65c21-istio-podinfo" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:54:34.038778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.038752    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-envoy" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:34.038867 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.038783    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-kube-api-access-lffc8" pod "e77765a7-6626-4005-b10c-db1068f65c21" (UID: "e77765a7-6626-4005-b10c-db1068f65c21"). InnerVolumeSpecName "kube-api-access-lffc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:34.130872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.130835    7019 generic.go:356] "Generic (PLEG): container finished" podID="e77765a7-6626-4005-b10c-db1068f65c21" containerID="b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d" exitCode=0
Nov 02 15:54:34.131058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.130894    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"
Nov 02 15:54:34.131058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.130927    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" event={"ID":"e77765a7-6626-4005-b10c-db1068f65c21","Type":"ContainerDied","Data":"b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d"}
Nov 02 15:54:34.131058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.130979    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc" event={"ID":"e77765a7-6626-4005-b10c-db1068f65c21","Type":"ContainerDied","Data":"90c3b255c92713703885a7ebea98d9eacb56d7c302226efcadec961e95d75daf"}
Nov 02 15:54:34.131058 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.131001    7019 scope.go:122] "RemoveContainer" containerID="b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d"
Nov 02 15:54:34.137817 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.137788    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e77765a7-6626-4005-b10c-db1068f65c21-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.137914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.137819    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lffc8\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-kube-api-access-lffc8\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.137914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.137834    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e77765a7-6626-4005-b10c-db1068f65c21-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.137914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.137842    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e77765a7-6626-4005-b10c-db1068f65c21-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.137914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.137850    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e77765a7-6626-4005-b10c-db1068f65c21-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:34.142698 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.142678    7019 scope.go:122] "RemoveContainer" containerID="d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891"
Nov 02 15:54:34.152654 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.152628    7019 scope.go:122] "RemoveContainer" containerID="b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d"
Nov 02 15:54:34.152904 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:34.152888    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d\": container with ID starting with b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d not found: ID does not exist" containerID="b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d"
Nov 02 15:54:34.152966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.152911    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d"} err="failed to get container status \"b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d\": rpc error: code = NotFound desc = could not find container \"b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d\": container with ID starting with b8fb3568e976ec948e3a1b4543e6b847103c30bb56ba42604de23c16046efb8d not found: ID does not exist"
Nov 02 15:54:34.152966 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.152926    7019 scope.go:122] "RemoveContainer" containerID="d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891"
Nov 02 15:54:34.153172 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:34.153153    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891\": container with ID starting with d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891 not found: ID does not exist" containerID="d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891"
Nov 02 15:54:34.153210 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.153178    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891"} err="failed to get container status \"d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891\": rpc error: code = NotFound desc = could not find container \"d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891\": container with ID starting with d5abb6166eb86f2a06d40a2ec7a9f045d8a036bc503121318ae04ab4d1d75891 not found: ID does not exist"
Nov 02 15:54:34.190604 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.190574    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"]
Nov 02 15:54:34.196584 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.196558    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8537e9-openshift-default-6bb675759-98dbc"]
Nov 02 15:54:34.900335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:34.900296    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:35.862115 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:35.862069    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77765a7-6626-4005-b10c-db1068f65c21" path="/var/lib/kubelet/pods/e77765a7-6626-4005-b10c-db1068f65c21/volumes"
Nov 02 15:54:36.332687 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:36.332584    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:36.809331 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:36.809293    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:37.067978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:37.067883    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"]
Nov 02 15:54:37.080710 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:37.080676    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-624254-hfjsk"]
Nov 02 15:54:37.082233 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:37.082192    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:37.499771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:37.499738    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:37.860816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:37.860756    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b14967b-0e2c-44da-a9a7-853d082ead96" path="/var/lib/kubelet/pods/3b14967b-0e2c-44da-a9a7-853d082ead96/volumes"
Nov 02 15:54:38.717470 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:38.717407    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:40.037748 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:40.037711    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:40.187902 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:40.187870    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:40.420342 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:40.420303    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:42.091714 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:42.091677    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:43.165969 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:43.165933    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:54:43.814127 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:43.814094    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:45.295505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:45.295464    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:45.814302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:45.814267    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:45.916236 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:45.916199    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"]
Nov 02 15:54:45.947435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:45.947398    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-fda646-hl84x"]
Nov 02 15:54:45.999380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:45.999348    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:46.221810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:46.221290    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"]
Nov 02 15:54:46.221810 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:46.221538    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" containerID="cri-o://b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de" gracePeriod=30
Nov 02 15:54:46.342998 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:46.342946    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:46.529681 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:46.529592    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:46.675345 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:46.675302    7019 patch_prober.go:29] interesting pod/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:54:46.675541 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:46.675367    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:54:47.823694 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:47.823641    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:47.860891 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:47.860858    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" path="/var/lib/kubelet/pods/6d3fec70-15ac-48fa-ac7d-ff7da5997d6c/volumes"
Nov 02 15:54:49.445385 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:49.445341    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:51.178425 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.178386    7019 scope.go:122] "RemoveContainer" containerID="c34f9261de8697f97e9fdf16b1b9fe38c9793629b892764d39ebae152a9c94b2"
Nov 02 15:54:51.189946 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.189920    7019 scope.go:122] "RemoveContainer" containerID="5bbf31be76d4c2d0ddfecedf9a25d4f4309497030895dcfd495a4dd57e072d7c"
Nov 02 15:54:51.305097 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.305059    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:51.505922 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.505892    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:51.522441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.522414    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:54:51.651055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651025    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-data\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651059    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-credential-socket\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651102    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-kube-api-access-nw5kv\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-kube-api-access-nw5kv\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651139    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dcb116-9946-41b3-9401-193076a9b410-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dcb116-9946-41b3-9401-193076a9b410-istiod-ca-cert\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651158    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-envoy\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651194    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0dcb116-9946-41b3-9401-193076a9b410-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dcb116-9946-41b3-9401-193076a9b410-istio-podinfo\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651231    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-certs\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651280    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-istio-token\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-istio-token\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651358    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-socket\") pod \"a0dcb116-9946-41b3-9401-193076a9b410\" (UID: \"a0dcb116-9946-41b3-9401-193076a9b410\") "
Nov 02 15:54:51.651690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651560    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-certs" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:51.651690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651590    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-credential-socket" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:51.651690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651597    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dcb116-9946-41b3-9401-193076a9b410-istiod-ca-cert" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:51.651884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651843    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.651884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651866    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-socket" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:51.652005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651871    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.652005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651909    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dcb116-9946-41b3-9401-193076a9b410-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.652005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.651931    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-data" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:51.653444 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.653410    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-envoy" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:51.653591 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.653571    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-istio-token" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:51.653887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.653856    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0dcb116-9946-41b3-9401-193076a9b410-istio-podinfo" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:54:51.653887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.653860    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-kube-api-access-nw5kv" pod "a0dcb116-9946-41b3-9401-193076a9b410" (UID: "a0dcb116-9946-41b3-9401-193076a9b410"). InnerVolumeSpecName "kube-api-access-nw5kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:51.740198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.740158    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:51.753483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.753420    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dcb116-9946-41b3-9401-193076a9b410-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.753483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.753443    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.753483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.753453    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.753483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.753461    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.753483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.753470    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw5kv\" (UniqueName: \"kubernetes.io/projected/a0dcb116-9946-41b3-9401-193076a9b410-kube-api-access-nw5kv\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.753483 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.753479    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dcb116-9946-41b3-9401-193076a9b410-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:51.990438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:51.990399    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:52.163575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.163531    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:52.269090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.269053    7019 generic.go:356] "Generic (PLEG): container finished" podID="a0dcb116-9946-41b3-9401-193076a9b410" containerID="b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de" exitCode=0
Nov 02 15:54:52.269675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.269099    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" event={"ID":"a0dcb116-9946-41b3-9401-193076a9b410","Type":"ContainerDied","Data":"b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de"}
Nov 02 15:54:52.269675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.269187    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9" event={"ID":"a0dcb116-9946-41b3-9401-193076a9b410","Type":"ContainerDied","Data":"ef8aba23a241af69ee7c3b17e3cec3f1ac498370141b2110c2dafe94eb2d8d06"}
Nov 02 15:54:52.269675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.269110    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"
Nov 02 15:54:52.269675 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.269223    7019 scope.go:122] "RemoveContainer" containerID="b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de"
Nov 02 15:54:52.279566 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.279538    7019 scope.go:122] "RemoveContainer" containerID="b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de"
Nov 02 15:54:52.279909 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:52.279886    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de\": container with ID starting with b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de not found: ID does not exist" containerID="b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de"
Nov 02 15:54:52.280025 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.279920    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de"} err="failed to get container status \"b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de\": rpc error: code = NotFound desc = could not find container \"b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de\": container with ID starting with b62e9b0f1d80941c8767cfdd731fa5af1de20c70495da8127a6a41034c7ea4de not found: ID does not exist"
Nov 02 15:54:52.332248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.332219    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"]
Nov 02 15:54:52.345011 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:52.344985    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-624254-openshift-default-7988c9d8b5-tt5h9"]
Nov 02 15:54:52.706435 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:52.706398    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:54:53.015760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:53.012426    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:53.843707 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:53.843671    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:53.861196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:53.861168    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dcb116-9946-41b3-9401-193076a9b410" path="/var/lib/kubelet/pods/a0dcb116-9946-41b3-9401-193076a9b410/volumes"
Nov 02 15:54:54.348978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:54.348940    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:54.501908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:54.501873    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:54.714308 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:54.714269    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:54.866741 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:54.866628    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:55.233246 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233215    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"]
Nov 02 15:54:55.233674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233633    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy"
Nov 02 15:54:55.233674 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233663    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy"
Nov 02 15:54:55.233761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233684    7019 cpu_manager.go:414] "containerMap: removing container" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy"
Nov 02 15:54:55.233761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233689    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy"
Nov 02 15:54:55.233761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233711    7019 cpu_manager.go:414] "containerMap: removing container" podUID="6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" containerName="revoke-keys"
Nov 02 15:54:55.233761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233718    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" containerName="revoke-keys"
Nov 02 15:54:55.233761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233727    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy"
Nov 02 15:54:55.233761 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233732    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy"
Nov 02 15:54:55.233956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233811    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy"
Nov 02 15:54:55.233956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233820    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77765a7-6626-4005-b10c-db1068f65c21" containerName="istio-proxy"
Nov 02 15:54:55.233956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233829    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3fec70-15ac-48fa-ac7d-ff7da5997d6c" containerName="revoke-keys"
Nov 02 15:54:55.233956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.233836    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dcb116-9946-41b3-9401-193076a9b410" containerName="istio-proxy"
Nov 02 15:54:55.238618 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.238596    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.260156 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.260131    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"]
Nov 02 15:54:55.270339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.270311    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:54:55.270459 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.270382    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:54:55.392119 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.392085    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.392309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.392150    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.392309 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.392253    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdt5m\" (UniqueName: \"kubernetes.io/projected/bfba16ea-e330-48f0-a558-be29d971abe0-kube-api-access-mdt5m\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.456274 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.456238    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"]
Nov 02 15:54:55.456460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.456431    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh" podUID="5c510c41-3a9e-441b-8114-83fc2bf2651c" containerName="discovery" containerID="cri-o://36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09" gracePeriod=30
Nov 02 15:54:55.494010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.493928    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.494151 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.494032    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.498101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.494098    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdt5m\" (UniqueName: \"kubernetes.io/projected/bfba16ea-e330-48f0-a558-be29d971abe0-kube-api-access-mdt5m\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.498101 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.495369    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.501595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.501569    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.518999 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.518976    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdt5m\" (UniqueName: \"kubernetes.io/projected/bfba16ea-e330-48f0-a558-be29d971abe0-kube-api-access-mdt5m\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.551771 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.551742    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:55.744147 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.744088    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:54:55.863728 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:54:55.863697    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfba16ea_e330_48f0_a558_be29d971abe0.slice/crio-63235719255b7ef6ce7244bb81301fc83201e0d18c104de8085245bba1fb3f45 WatchSource:0}: Error finding container 63235719255b7ef6ce7244bb81301fc83201e0d18c104de8085245bba1fb3f45: Status 404 returned error can't find the container with id 63235719255b7ef6ce7244bb81301fc83201e0d18c104de8085245bba1fb3f45
Nov 02 15:54:55.870062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.870041    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"]
Nov 02 15:54:55.899978 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.898739    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-dns-cert\") pod \"5c510c41-3a9e-441b-8114-83fc2bf2651c\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") "
Nov 02 15:54:55.901435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.900094    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-kubeconfig\") pod \"5c510c41-3a9e-441b-8114-83fc2bf2651c\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") "
Nov 02 15:54:55.901435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.900698    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-cacerts\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-cacerts\") pod \"5c510c41-3a9e-441b-8114-83fc2bf2651c\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") "
Nov 02 15:54:55.901435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.901239    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-ca-configmap\") pod \"5c510c41-3a9e-441b-8114-83fc2bf2651c\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") "
Nov 02 15:54:55.901435 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.901291    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c510c41-3a9e-441b-8114-83fc2bf2651c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c510c41-3a9e-441b-8114-83fc2bf2651c-local-certs\") pod \"5c510c41-3a9e-441b-8114-83fc2bf2651c\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") "
Nov 02 15:54:55.901809 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.901783    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-kube-api-access-7zhgt\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-kube-api-access-7zhgt\") pod \"5c510c41-3a9e-441b-8114-83fc2bf2651c\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") "
Nov 02 15:54:55.901879 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.901826    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-token\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-token\") pod \"5c510c41-3a9e-441b-8114-83fc2bf2651c\" (UID: \"5c510c41-3a9e-441b-8114-83fc2bf2651c\") "
Nov 02 15:54:55.902569 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.902545    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-dns-cert" pod "5c510c41-3a9e-441b-8114-83fc2bf2651c" (UID: "5c510c41-3a9e-441b-8114-83fc2bf2651c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:55.903120 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.903093    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-ca-configmap" pod "5c510c41-3a9e-441b-8114-83fc2bf2651c" (UID: "5c510c41-3a9e-441b-8114-83fc2bf2651c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:55.904312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.904284    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-cacerts" pod "5c510c41-3a9e-441b-8114-83fc2bf2651c" (UID: "5c510c41-3a9e-441b-8114-83fc2bf2651c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:55.904804 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.904780    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-kubeconfig" pod "5c510c41-3a9e-441b-8114-83fc2bf2651c" (UID: "5c510c41-3a9e-441b-8114-83fc2bf2651c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:55.904932 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.904821    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-token" pod "5c510c41-3a9e-441b-8114-83fc2bf2651c" (UID: "5c510c41-3a9e-441b-8114-83fc2bf2651c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:55.905005 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.904935    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-kube-api-access-7zhgt" pod "5c510c41-3a9e-441b-8114-83fc2bf2651c" (UID: "5c510c41-3a9e-441b-8114-83fc2bf2651c"). InnerVolumeSpecName "kube-api-access-7zhgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:55.905393 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:55.905376    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c510c41-3a9e-441b-8114-83fc2bf2651c-local-certs" pod "5c510c41-3a9e-441b-8114-83fc2bf2651c" (UID: "5c510c41-3a9e-441b-8114-83fc2bf2651c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:54:56.003441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.003343    7019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-dns-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:56.003441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.003376    7019 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-kubeconfig\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:56.003441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.003392    7019 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c510c41-3a9e-441b-8114-83fc2bf2651c-cacerts\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:56.003441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.003402    7019 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-csr-ca-configmap\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:56.003441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.003412    7019 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c510c41-3a9e-441b-8114-83fc2bf2651c-local-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:56.003441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.003421    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhgt\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-kube-api-access-7zhgt\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:56.003441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.003429    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c510c41-3a9e-441b-8114-83fc2bf2651c-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:56.225209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.225171    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:56.301105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.301011    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz" event={"ID":"bfba16ea-e330-48f0-a558-be29d971abe0","Type":"ContainerStarted","Data":"24ae1afef37f12921f805778eefbff0260e2033d0042443ca76a66904b886039"}
Nov 02 15:54:56.301105 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.301085    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz" event={"ID":"bfba16ea-e330-48f0-a558-be29d971abe0","Type":"ContainerStarted","Data":"63235719255b7ef6ce7244bb81301fc83201e0d18c104de8085245bba1fb3f45"}
Nov 02 15:54:56.302605 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.302578    7019 generic.go:356] "Generic (PLEG): container finished" podID="5c510c41-3a9e-441b-8114-83fc2bf2651c" containerID="36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09" exitCode=0
Nov 02 15:54:56.302760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.302610    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"
Nov 02 15:54:56.302760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.302627    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh" event={"ID":"5c510c41-3a9e-441b-8114-83fc2bf2651c","Type":"ContainerDied","Data":"36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09"}
Nov 02 15:54:56.302760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.302694    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh" event={"ID":"5c510c41-3a9e-441b-8114-83fc2bf2651c","Type":"ContainerDied","Data":"492c327bbbb7f6514570ed1707e59fb1e93f3143d926a98e3f66ad042f4ba14f"}
Nov 02 15:54:56.302760 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.302723    7019 scope.go:122] "RemoveContainer" containerID="36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09"
Nov 02 15:54:56.315615 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.315591    7019 scope.go:122] "RemoveContainer" containerID="36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09"
Nov 02 15:54:56.315924 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:54:56.315904    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09\": container with ID starting with 36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09 not found: ID does not exist" containerID="36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09"
Nov 02 15:54:56.316012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.315934    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09"} err="failed to get container status \"36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09\": rpc error: code = NotFound desc = could not find container \"36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09\": container with ID starting with 36ea3d6403a99e5adb6fbf27abb29234a941def68bb126cc2b799a48f138fd09 not found: ID does not exist"
Nov 02 15:54:56.344563 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.344512    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz" podStartSLOduration=1.344500297 podStartE2EDuration="1.344500297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:54:56.343200531 +0000 UTC m=+3490.597154152" watchObservedRunningTime="2026-11-02 15:54:56.344500297 +0000 UTC m=+3490.598453958"
Nov 02 15:54:56.385372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.385341    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"]
Nov 02 15:54:56.397943 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.397913    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-j7snh"]
Nov 02 15:54:56.449527 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.449494    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:56.867797 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:56.867764    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:54:57.312334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:57.312285    7019 generic.go:356] "Generic (PLEG): container finished" podID="bfba16ea-e330-48f0-a558-be29d971abe0" containerID="24ae1afef37f12921f805778eefbff0260e2033d0042443ca76a66904b886039" exitCode=0
Nov 02 15:54:57.312334 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:57.312334    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz" event={"ID":"bfba16ea-e330-48f0-a558-be29d971abe0","Type":"ContainerDied","Data":"24ae1afef37f12921f805778eefbff0260e2033d0042443ca76a66904b886039"}
Nov 02 15:54:57.860692 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:57.860635    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c510c41-3a9e-441b-8114-83fc2bf2651c" path="/var/lib/kubelet/pods/5c510c41-3a9e-441b-8114-83fc2bf2651c/volumes"
Nov 02 15:54:58.220322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.220237    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:54:58.457659 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.457618    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:58.631931 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.631899    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfba16ea-e330-48f0-a558-be29d971abe0-kube-api-access-mdt5m\" (UniqueName: \"kubernetes.io/projected/bfba16ea-e330-48f0-a558-be29d971abe0-kube-api-access-mdt5m\") pod \"bfba16ea-e330-48f0-a558-be29d971abe0\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") "
Nov 02 15:54:58.632088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.631983    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-xks-ca\") pod \"bfba16ea-e330-48f0-a558-be29d971abe0\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") "
Nov 02 15:54:58.632088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.632034    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-openshift-ca\") pod \"bfba16ea-e330-48f0-a558-be29d971abe0\" (UID: \"bfba16ea-e330-48f0-a558-be29d971abe0\") "
Nov 02 15:54:58.632426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.632401    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-openshift-ca" pod "bfba16ea-e330-48f0-a558-be29d971abe0" (UID: "bfba16ea-e330-48f0-a558-be29d971abe0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:54:58.634082 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.634062    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-xks-ca" pod "bfba16ea-e330-48f0-a558-be29d971abe0" (UID: "bfba16ea-e330-48f0-a558-be29d971abe0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:54:58.634196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.634180    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfba16ea-e330-48f0-a558-be29d971abe0-kube-api-access-mdt5m" pod "bfba16ea-e330-48f0-a558-be29d971abe0" (UID: "bfba16ea-e330-48f0-a558-be29d971abe0"). InnerVolumeSpecName "kube-api-access-mdt5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:54:58.733344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.733308    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:58.733344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.733339    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdt5m\" (UniqueName: \"kubernetes.io/projected/bfba16ea-e330-48f0-a558-be29d971abe0-kube-api-access-mdt5m\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:58.733344 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:58.733350    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfba16ea-e330-48f0-a558-be29d971abe0-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:54:59.326957 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:59.326928    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"
Nov 02 15:54:59.327177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:59.326970    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz" event={"ID":"bfba16ea-e330-48f0-a558-be29d971abe0","Type":"ContainerDied","Data":"63235719255b7ef6ce7244bb81301fc83201e0d18c104de8085245bba1fb3f45"}
Nov 02 15:54:59.327177 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:59.327019    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63235719255b7ef6ce7244bb81301fc83201e0d18c104de8085245bba1fb3f45"
Nov 02 15:54:59.916916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:54:59.916882    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:00.996219 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:00.996182    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:01.594165 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:01.594129    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:03.523801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:03.523763    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:05.107688 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:05.107638    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:05.903925 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:05.903896    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:06.048798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:06.048761    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:06.778186 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:06.778147    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:08.154315 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:08.154273    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:08.652606 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:08.652564    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:08.753383 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:08.753340    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"]
Nov 02 15:55:08.766107 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:08.766070    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-cce791-hrkgz"]
Nov 02 15:55:08.800454 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:08.800422    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:09.293303 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:09.293270    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:09.788426 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:09.788393    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:09.861028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:09.860992    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfba16ea-e330-48f0-a558-be29d971abe0" path="/var/lib/kubelet/pods/bfba16ea-e330-48f0-a558-be29d971abe0/volumes"
Nov 02 15:55:10.490478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:10.490439    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:11.377858 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:11.377820    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:55:12.190884 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:12.190845    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:13.787261 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:13.787219    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:14.866721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:14.866644    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:15.499438 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:15.499399    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:16.203296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:16.203251    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:16.357347 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:16.357310    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:16.500466 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:16.500372    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"]
Nov 02 15:55:16.500640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:16.500602    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy" containerID="cri-o://781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe" gracePeriod=30
Nov 02 15:55:16.573599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:16.573564    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:17.082561 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:17.082512    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:17.242484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:17.242448    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:18.809092 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:18.809048    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:20.403405 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:20.403366    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:21.753409 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.753382    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:55:21.871141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871115    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-podinfo\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871154    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-credential-socket\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871179    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istiod-ca-cert\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871202    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-envoy\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871317    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-socket\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871354    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-data\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871394    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-certs\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871460 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871424    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-token\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-token\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871464    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-kube-api-access-4nx98\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-kube-api-access-4nx98\") pod \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\" (UID: \"93149e3b-33b5-4cce-bbec-972ffd4b1c61\") "
Nov 02 15:55:21.871911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871464    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-credential-socket" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.871911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871687    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-data" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.871911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871705    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-socket" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.871911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871723    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-certs" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.872192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871963    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.872192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871984    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.872192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.871999    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.872192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.872013    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.872192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.872173    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istiod-ca-cert" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:21.873589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.873561    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-kube-api-access-4nx98" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "kube-api-access-4nx98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:21.873589 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.873572    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-envoy" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:21.873869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.873846    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-podinfo" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:55:21.873869 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.873860    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-token" pod "93149e3b-33b5-4cce-bbec-972ffd4b1c61" (UID: "93149e3b-33b5-4cce-bbec-972ffd4b1c61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:21.972833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.972799    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.972833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.972826    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nx98\" (UniqueName: \"kubernetes.io/projected/93149e3b-33b5-4cce-bbec-972ffd4b1c61-kube-api-access-4nx98\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.972833 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.972836    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.973075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.972847    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:21.973075 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:21.972855    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93149e3b-33b5-4cce-bbec-972ffd4b1c61-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:22.002455 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.002426    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:22.040875 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.040838    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"]
Nov 02 15:55:22.191116 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.191010    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:22.493050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.492971    7019 generic.go:356] "Generic (PLEG): container finished" podID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerID="781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe" exitCode=0
Nov 02 15:55:22.493050 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.493011    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" event={"ID":"93149e3b-33b5-4cce-bbec-972ffd4b1c61","Type":"ContainerDied","Data":"781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe"}
Nov 02 15:55:22.493285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.493060    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"
Nov 02 15:55:22.493285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.493075    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k" event={"ID":"93149e3b-33b5-4cce-bbec-972ffd4b1c61","Type":"ContainerDied","Data":"629ac0a57ef477dfd7e5f14497fd8bf0600357f755c58fc696f1c1525709d81f"}
Nov 02 15:55:22.493285 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.493097    7019 scope.go:122] "RemoveContainer" containerID="781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe"
Nov 02 15:55:22.504491 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.504473    7019 scope.go:122] "RemoveContainer" containerID="1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda"
Nov 02 15:55:22.513312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.513295    7019 scope.go:122] "RemoveContainer" containerID="781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe"
Nov 02 15:55:22.513553 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:55:22.513533    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe\": container with ID starting with 781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe not found: ID does not exist" containerID="781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe"
Nov 02 15:55:22.513620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.513560    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe"} err="failed to get container status \"781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe\": rpc error: code = NotFound desc = could not find container \"781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe\": container with ID starting with 781374755e61b859a05f915e40e17cac3b4699c800595221a5cfa3ff35e98cfe not found: ID does not exist"
Nov 02 15:55:22.513620 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.513575    7019 scope.go:122] "RemoveContainer" containerID="1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda"
Nov 02 15:55:22.513819 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:55:22.513793    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda\": container with ID starting with 1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda not found: ID does not exist" containerID="1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda"
Nov 02 15:55:22.513876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.513825    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda"} err="failed to get container status \"1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda\": rpc error: code = NotFound desc = could not find container \"1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda\": container with ID starting with 1a7b87324cfc46a6bfb6252ce6da32c5cf7c8b8bc949a597492546024416dfda not found: ID does not exist"
Nov 02 15:55:22.547006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.546978    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"]
Nov 02 15:55:22.560114 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:22.560090    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-cce791-openshift-default-c54fc95f9-xbk8k"]
Nov 02 15:55:22.704876 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:55:22.704840    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:55:23.684827 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:23.684796    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:23.861326 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:23.861288    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" path="/var/lib/kubelet/pods/93149e3b-33b5-4cce-bbec-972ffd4b1c61/volumes"
Nov 02 15:55:25.282028 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:25.281991    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:26.793855 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:26.793813    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:28.043795 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.043760    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:28.185198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185166    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"]
Nov 02 15:55:28.185590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185577    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5c510c41-3a9e-441b-8114-83fc2bf2651c" containerName="discovery"
Nov 02 15:55:28.185643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185591    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c510c41-3a9e-441b-8114-83fc2bf2651c" containerName="discovery"
Nov 02 15:55:28.185643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185609    7019 cpu_manager.go:414] "containerMap: removing container" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy"
Nov 02 15:55:28.185643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185615    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy"
Nov 02 15:55:28.185643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185623    7019 cpu_manager.go:414] "containerMap: removing container" podUID="bfba16ea-e330-48f0-a558-be29d971abe0" containerName="revoke-keys"
Nov 02 15:55:28.185643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185628    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfba16ea-e330-48f0-a558-be29d971abe0" containerName="revoke-keys"
Nov 02 15:55:28.185643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185656    7019 cpu_manager.go:414] "containerMap: removing container" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy"
Nov 02 15:55:28.185643 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185661    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy"
Nov 02 15:55:28.185908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185740    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy"
Nov 02 15:55:28.185908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185747    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfba16ea-e330-48f0-a558-be29d971abe0" containerName="revoke-keys"
Nov 02 15:55:28.185908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.185763    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c510c41-3a9e-441b-8114-83fc2bf2651c" containerName="discovery"
Nov 02 15:55:28.190433 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.190408    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.209230 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.209186    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:28.211248 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.211226    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"]
Nov 02 15:55:28.292983 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.292951    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:55:28.293141 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.292996    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 15:55:28.342753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.342673    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.342753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.342743    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.342952 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.342832    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4kn\" (UniqueName: \"kubernetes.io/projected/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-kube-api-access-8r4kn\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.443481 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.443448    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.443660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.443496    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4kn\" (UniqueName: \"kubernetes.io/projected/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-kube-api-access-8r4kn\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.443660 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.443569    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.444370 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.444336    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.445811 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.445782    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.467908 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.467878    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4kn\" (UniqueName: \"kubernetes.io/projected/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-kube-api-access-8r4kn\") pod \"maas-api-revoke-sub-0c911da5-sqvzq\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.507697 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.507639    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:28.819046 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:28.819020    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"]
Nov 02 15:55:28.819162 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:55:28.819059    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f38c348_7d9b_4e3c_ba2f_b4d6d75ba0e2.slice/crio-d2515bc9b57500fde75e5f54d28d095b41ac8ae607d2665c524bcc58be84bf51 WatchSource:0}: Error finding container d2515bc9b57500fde75e5f54d28d095b41ac8ae607d2665c524bcc58be84bf51: Status 404 returned error can't find the container with id d2515bc9b57500fde75e5f54d28d095b41ac8ae607d2665c524bcc58be84bf51
Nov 02 15:55:29.543293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:29.543258    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq" event={"ID":"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2","Type":"ContainerStarted","Data":"920a0d4e4b14bfe433aec5634da25cb613fba7530db656de56cb78489d5a305e"}
Nov 02 15:55:29.543799 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:29.543316    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq" event={"ID":"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2","Type":"ContainerStarted","Data":"d2515bc9b57500fde75e5f54d28d095b41ac8ae607d2665c524bcc58be84bf51"}
Nov 02 15:55:29.587149 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:29.587083    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq" podStartSLOduration=1.587070315 podStartE2EDuration="1.587070315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:55:29.585318043 +0000 UTC m=+3523.839271668" watchObservedRunningTime="2026-11-02 15:55:29.587070315 +0000 UTC m=+3523.841023978"
Nov 02 15:55:30.552205 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:30.552166    7019 generic.go:356] "Generic (PLEG): container finished" podID="3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" containerID="920a0d4e4b14bfe433aec5634da25cb613fba7530db656de56cb78489d5a305e" exitCode=0
Nov 02 15:55:30.552627 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:30.552255    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq" event={"ID":"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2","Type":"ContainerDied","Data":"920a0d4e4b14bfe433aec5634da25cb613fba7530db656de56cb78489d5a305e"}
Nov 02 15:55:31.689887 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.689858    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:31.877507 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.877470    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-xks-ca\") pod \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") "
Nov 02 15:55:31.877729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.877588    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-openshift-ca\") pod \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") "
Nov 02 15:55:31.877729 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.877666    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-kube-api-access-8r4kn\" (UniqueName: \"kubernetes.io/projected/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-kube-api-access-8r4kn\") pod \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\" (UID: \"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2\") "
Nov 02 15:55:31.877997 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.877969    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-openshift-ca" pod "3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" (UID: "3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:55:31.879595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.879573    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-xks-ca" pod "3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" (UID: "3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:31.879851 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.879830    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-kube-api-access-8r4kn" pod "3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" (UID: "3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2"). InnerVolumeSpecName "kube-api-access-8r4kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:31.979267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.979219    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.979267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.979258    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:31.979267 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:31.979267    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r4kn\" (UniqueName: \"kubernetes.io/projected/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2-kube-api-access-8r4kn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:32.565571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:32.565540    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"
Nov 02 15:55:32.565571 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:32.565561    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq" event={"ID":"3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2","Type":"ContainerDied","Data":"d2515bc9b57500fde75e5f54d28d095b41ac8ae607d2665c524bcc58be84bf51"}
Nov 02 15:55:32.565874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:32.565602    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2515bc9b57500fde75e5f54d28d095b41ac8ae607d2665c524bcc58be84bf51"
Nov 02 15:55:37.061778 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.061719    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main" containerID="cri-o://7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de" gracePeriod=45
Nov 02 15:55:37.321328 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.321262    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:55:37.331126 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331099    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-dshm\") pod \"af87adc8-9bc6-4766-9850-45cb1e77469b\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") "
Nov 02 15:55:37.331231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331148    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-home\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-home\") pod \"af87adc8-9bc6-4766-9850-45cb1e77469b\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") "
Nov 02 15:55:37.331231 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331183    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af87adc8-9bc6-4766-9850-45cb1e77469b-kube-api-access-r5lhp\" (UniqueName: \"kubernetes.io/projected/af87adc8-9bc6-4766-9850-45cb1e77469b-kube-api-access-r5lhp\") pod \"af87adc8-9bc6-4766-9850-45cb1e77469b\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") "
Nov 02 15:55:37.331360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331253    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-tmp-dir\") pod \"af87adc8-9bc6-4766-9850-45cb1e77469b\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") "
Nov 02 15:55:37.331360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331298    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af87adc8-9bc6-4766-9850-45cb1e77469b-tls-certs\" (UniqueName: \"kubernetes.io/secret/af87adc8-9bc6-4766-9850-45cb1e77469b-tls-certs\") pod \"af87adc8-9bc6-4766-9850-45cb1e77469b\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") "
Nov 02 15:55:37.331480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331402    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-model-cache\") pod \"af87adc8-9bc6-4766-9850-45cb1e77469b\" (UID: \"af87adc8-9bc6-4766-9850-45cb1e77469b\") "
Nov 02 15:55:37.331546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331448    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-home" pod "af87adc8-9bc6-4766-9850-45cb1e77469b" (UID: "af87adc8-9bc6-4766-9850-45cb1e77469b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.331546 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331519    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-tmp-dir" pod "af87adc8-9bc6-4766-9850-45cb1e77469b" (UID: "af87adc8-9bc6-4766-9850-45cb1e77469b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.331774 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331753    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-model-cache" pod "af87adc8-9bc6-4766-9850-45cb1e77469b" (UID: "af87adc8-9bc6-4766-9850-45cb1e77469b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.331893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.331869    7019 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-model-cache\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.332055 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.332034    7019 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-home\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.332150 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.332060    7019 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-tmp-dir\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.333484 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.333461    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-dshm" pod "af87adc8-9bc6-4766-9850-45cb1e77469b" (UID: "af87adc8-9bc6-4766-9850-45cb1e77469b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:55:37.333573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.333517    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af87adc8-9bc6-4766-9850-45cb1e77469b-tls-certs" pod "af87adc8-9bc6-4766-9850-45cb1e77469b" (UID: "af87adc8-9bc6-4766-9850-45cb1e77469b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:55:37.333573 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.333551    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af87adc8-9bc6-4766-9850-45cb1e77469b-kube-api-access-r5lhp" pod "af87adc8-9bc6-4766-9850-45cb1e77469b" (UID: "af87adc8-9bc6-4766-9850-45cb1e77469b"). InnerVolumeSpecName "kube-api-access-r5lhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:55:37.432595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.432560    7019 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af87adc8-9bc6-4766-9850-45cb1e77469b-dshm\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.432595 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.432594    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5lhp\" (UniqueName: \"kubernetes.io/projected/af87adc8-9bc6-4766-9850-45cb1e77469b-kube-api-access-r5lhp\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.432821 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.432607    7019 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af87adc8-9bc6-4766-9850-45cb1e77469b-tls-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:55:37.604201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.604095    7019 generic.go:356] "Generic (PLEG): container finished" podID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerID="7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de" exitCode=0
Nov 02 15:55:37.604201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.604159    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"
Nov 02 15:55:37.604201 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.604186    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" event={"ID":"af87adc8-9bc6-4766-9850-45cb1e77469b","Type":"ContainerDied","Data":"7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de"}
Nov 02 15:55:37.604500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.604229    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz" event={"ID":"af87adc8-9bc6-4766-9850-45cb1e77469b","Type":"ContainerDied","Data":"feb9acb9907cbd3c58b10a5a9c44bffba6361d90dc542af1e670759ef6e8f83c"}
Nov 02 15:55:37.604500 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.604246    7019 scope.go:122] "RemoveContainer" containerID="7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de"
Nov 02 15:55:37.615103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.615078    7019 scope.go:122] "RemoveContainer" containerID="7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de"
Nov 02 15:55:37.615375 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:55:37.615356    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de\": container with ID starting with 7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de not found: ID does not exist" containerID="7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de"
Nov 02 15:55:37.615439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.615383    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de"} err="failed to get container status \"7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de\": rpc error: code = NotFound desc = could not find container \"7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de\": container with ID starting with 7aa929bb60b2c65fac0c8a3cd81fbd54e2d318d34968e14b7e02648f980784de not found: ID does not exist"
Nov 02 15:55:37.660094 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.660061    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"]
Nov 02 15:55:37.672854 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.672828    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5dd8e9/distinct-w1-5dd8e9-kserve-69cd4cc97f-8z2cz"]
Nov 02 15:55:37.860577 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:37.860500    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" path="/var/lib/kubelet/pods/af87adc8-9bc6-4766-9850-45cb1e77469b/volumes"
Nov 02 15:55:40.547683 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:40.547638    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:55:43.384270 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:43.384236    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:44.638480 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:44.638432    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:47.622211 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:47.622170    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:55:51.246785 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:51.246750    7019 scope.go:122] "RemoveContainer" containerID="2adcd2af02395ed6cebe8475aaa00e4db1f3a09e658dcec5083a3e665ee9571b"
Nov 02 15:55:51.256195 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:51.256172    7019 scope.go:122] "RemoveContainer" containerID="2c94406c7fd744bf9286ffe37be08fcd3f4b0d9d232e46699def1d2d98b2ce91"
Nov 02 15:55:51.264502 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:51.264483    7019 scope.go:122] "RemoveContainer" containerID="5f896f35117cb7db74d850a41b52f0595b6426b3aee997fcfbba4eafa305f5ba"
Nov 02 15:55:51.275109 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:51.275085    7019 scope.go:122] "RemoveContainer" containerID="f73369b86d50ebee2b120626e21804b579fa2f6da28b2640820b729036974d75"
Nov 02 15:55:52.707666 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:55:52.707616    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:55:54.471167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:54.471133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:56.381073 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:56.381035    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:58.898641 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:58.898602    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:55:59.048847 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:55:59.048813    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:00.314523 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:00.314486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:01.790367 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:01.790321    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:01.990816 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:01.990770    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:02.121599 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:02.121563    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"]
Nov 02 15:56:02.121834 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:02.121811    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy" containerID="cri-o://966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8" gracePeriod=30
Nov 02 15:56:02.199871 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:02.199832    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:06.231038 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:06.230997    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:07.387140 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.387111    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:56:07.551251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551161    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab090f8-be90-4abf-b504-35919d67bb0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab090f8-be90-4abf-b504-35919d67bb0a-istiod-ca-cert\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551251 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551238    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-kube-api-access-xplcd\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-kube-api-access-xplcd\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551287    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-credential-socket\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551319    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-data\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551387    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-envoy\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551428    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dab090f8-be90-4abf-b504-35919d67bb0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab090f8-be90-4abf-b504-35919d67bb0a-istio-podinfo\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551482 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551456    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-certs\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551484    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-istio-token\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-istio-token\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551526    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-socket\") pod \"dab090f8-be90-4abf-b504-35919d67bb0a\" (UID: \"dab090f8-be90-4abf-b504-35919d67bb0a\") "
Nov 02 15:56:07.551798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551576    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-credential-socket" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:56:07.551798 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551619    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-data" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:56:07.552006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551864    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-certs" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:56:07.552006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551978    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-socket" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:56:07.552006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551993    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.552006 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.551998    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab090f8-be90-4abf-b504-35919d67bb0a-istiod-ca-cert" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:56:07.552198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.552012    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.552198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.552026    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.554062 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.554024    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-kube-api-access-xplcd" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "kube-api-access-xplcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:56:07.554196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.554076    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-envoy" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:56:07.554196 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.554184    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-istio-token" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:56:07.554321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.554186    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dab090f8-be90-4abf-b504-35919d67bb0a-istio-podinfo" pod "dab090f8-be90-4abf-b504-35919d67bb0a" (UID: "dab090f8-be90-4abf-b504-35919d67bb0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:56:07.652939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.652909    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dab090f8-be90-4abf-b504-35919d67bb0a-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.652939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.652935    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xplcd\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-kube-api-access-xplcd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.652939 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.652947    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.653191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.652955    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dab090f8-be90-4abf-b504-35919d67bb0a-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.653191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.652965    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dab090f8-be90-4abf-b504-35919d67bb0a-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.653191 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.652974    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dab090f8-be90-4abf-b504-35919d67bb0a-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 15:56:07.805617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.805520    7019 generic.go:356] "Generic (PLEG): container finished" podID="dab090f8-be90-4abf-b504-35919d67bb0a" containerID="966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8" exitCode=0
Nov 02 15:56:07.805617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.805582    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"
Nov 02 15:56:07.805617 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.805598    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" event={"ID":"dab090f8-be90-4abf-b504-35919d67bb0a","Type":"ContainerDied","Data":"966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8"}
Nov 02 15:56:07.805929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.805672    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w" event={"ID":"dab090f8-be90-4abf-b504-35919d67bb0a","Type":"ContainerDied","Data":"cb8315a39ed0e84532f12b1e8463cbd1a1bbcf2279deef4873221d00df5cb68c"}
Nov 02 15:56:07.805929 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.805708    7019 scope.go:122] "RemoveContainer" containerID="966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8"
Nov 02 15:56:07.817079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.816967    7019 scope.go:122] "RemoveContainer" containerID="966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8"
Nov 02 15:56:07.817914 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:56:07.817291    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8\": container with ID starting with 966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8 not found: ID does not exist" containerID="966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8"
Nov 02 15:56:07.817914 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.817330    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8"} err="failed to get container status \"966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8\": rpc error: code = NotFound desc = could not find container \"966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8\": container with ID starting with 966385a80b9f62a231526e494cd76959e992a140cb727e0d91b7dce3a76109c8 not found: ID does not exist"
Nov 02 15:56:07.881883 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.881848    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"]
Nov 02 15:56:07.893558 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:07.893525    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5dd8e9-openshift-default-84b45c6965-twc6w"]
Nov 02 15:56:09.867511 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:09.867471    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" path="/var/lib/kubelet/pods/dab090f8-be90-4abf-b504-35919d67bb0a/volumes"
Nov 02 15:56:09.920250 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:09.920218    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:14.032090 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:14.032053    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"]
Nov 02 15:56:14.045014 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:14.044982    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"]
Nov 02 15:56:14.046372 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:14.046358    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e2d6b4-4mctf"]
Nov 02 15:56:14.047721 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:14.047702    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ecad19-6fzsk"]
Nov 02 15:56:14.986868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:14.986833    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:15.860953 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:15.860920    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334660bb-1214-473e-b530-84d56a927b42" path="/var/lib/kubelet/pods/334660bb-1214-473e-b530-84d56a927b42/volumes"
Nov 02 15:56:15.861366 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:15.861336    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cbf4c2d-fecb-4bf2-9c87-71ab236580f0" path="/var/lib/kubelet/pods/7cbf4c2d-fecb-4bf2-9c87-71ab236580f0/volumes"
Nov 02 15:56:22.703488 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:56:22.703451    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:56:32.077008 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:32.076972    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:32.242012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:32.241979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:56:43.925099 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925059    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"]
Nov 02 15:56:43.925610 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925588    7019 cpu_manager.go:414] "containerMap: removing container" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main"
Nov 02 15:56:43.925695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925614    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main"
Nov 02 15:56:43.925695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925642    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy"
Nov 02 15:56:43.925695 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925673    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy"
Nov 02 15:56:43.925813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925711    7019 cpu_manager.go:414] "containerMap: removing container" podUID="3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" containerName="revoke-keys"
Nov 02 15:56:43.925813 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925719    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" containerName="revoke-keys"
Nov 02 15:56:43.925894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925817    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="93149e3b-33b5-4cce-bbec-972ffd4b1c61" containerName="istio-proxy"
Nov 02 15:56:43.925894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925829    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="af87adc8-9bc6-4766-9850-45cb1e77469b" containerName="main"
Nov 02 15:56:43.925894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925838    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab090f8-be90-4abf-b504-35919d67bb0a" containerName="istio-proxy"
Nov 02 15:56:43.925894 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.925847    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" containerName="revoke-keys"
Nov 02 15:56:43.928753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.928736    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:43.959084 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:43.959051    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"]
Nov 02 15:56:44.094209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094157    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094209 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094208    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094231    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094316    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094396    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094496    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094557    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094637    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbgtg\" (UniqueName: \"kubernetes.io/projected/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-kube-api-access-tbgtg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.094712 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.094695    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195449    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbgtg\" (UniqueName: \"kubernetes.io/projected/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-kube-api-access-tbgtg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195497    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195520    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195551 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195537    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195678    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195720    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195804    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195852    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.195962 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.195883    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.196273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.196038    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.196273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.196151    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.196273 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.196242    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.196387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.196290    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.196387 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.196337    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.198031 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.198006    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.198335 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.198313    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.220288 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.220260    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.220439 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.220420    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbgtg\" (UniqueName: \"kubernetes.io/projected/74ffc3f9-8341-41be-8b15-c9829dfb6fb0-kube-api-access-tbgtg\") pod \"maas-default-gateway-openshift-default-fcb75c45f-8l7bs\" (UID: \"74ffc3f9-8341-41be-8b15-c9829dfb6fb0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.242380 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.242351    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:44.562630 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.562599    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"]
Nov 02 15:56:44.563620 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:56:44.563598    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ffc3f9_8341_41be_8b15_c9829dfb6fb0.slice/crio-54b60b135225e29141bdbcecdd3313a3d0b9c0b1021cf3e560a0eec6739fc353 WatchSource:0}: Error finding container 54b60b135225e29141bdbcecdd3313a3d0b9c0b1021cf3e560a0eec6739fc353: Status 404 returned error can't find the container with id 54b60b135225e29141bdbcecdd3313a3d0b9c0b1021cf3e560a0eec6739fc353
Nov 02 15:56:44.565786 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.565755    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:56:44.565856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.565819    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:56:44.565856 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:44.565849    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:56:45.044634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:45.044592    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" event={"ID":"74ffc3f9-8341-41be-8b15-c9829dfb6fb0","Type":"ContainerStarted","Data":"6c0609790c620ea9975afbaf5715c1b25fb9a946fb2d875cca29e444238ddc75"}
Nov 02 15:56:45.045155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:45.044711    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" event={"ID":"74ffc3f9-8341-41be-8b15-c9829dfb6fb0","Type":"ContainerStarted","Data":"54b60b135225e29141bdbcecdd3313a3d0b9c0b1021cf3e560a0eec6739fc353"}
Nov 02 15:56:45.112010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:45.111949    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podStartSLOduration=2.111936569 podStartE2EDuration="2.111936569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:56:45.109922346 +0000 UTC m=+3599.363875962" watchObservedRunningTime="2026-11-02 15:56:45.111936569 +0000 UTC m=+3599.365890193"
Nov 02 15:56:45.243360 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:45.243319    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:45.244709 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:45.244684    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8l7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body=
Nov 02 15:56:45.244835 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:45.244735    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podUID="74ffc3f9-8341-41be-8b15-c9829dfb6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused"
Nov 02 15:56:46.243223 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:46.243185    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8l7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body=
Nov 02 15:56:46.243635 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:46.243238    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podUID="74ffc3f9-8341-41be-8b15-c9829dfb6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused"
Nov 02 15:56:47.242870 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:47.242833    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8l7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body=
Nov 02 15:56:47.243111 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:47.242902    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podUID="74ffc3f9-8341-41be-8b15-c9829dfb6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused"
Nov 02 15:56:48.243128 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:48.243088    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8l7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body=
Nov 02 15:56:48.243640 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:48.243150    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podUID="74ffc3f9-8341-41be-8b15-c9829dfb6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused"
Nov 02 15:56:49.242985 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:49.242949    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8l7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body=
Nov 02 15:56:49.243174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:49.243003    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podUID="74ffc3f9-8341-41be-8b15-c9829dfb6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused"
Nov 02 15:56:50.243332 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:50.243285    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8l7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body=
Nov 02 15:56:50.243781 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:50.243352    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podUID="74ffc3f9-8341-41be-8b15-c9829dfb6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused"
Nov 02 15:56:51.243304 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:51.243268    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-8l7bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body=
Nov 02 15:56:51.243503 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:51.243329    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs" podUID="74ffc3f9-8341-41be-8b15-c9829dfb6fb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused"
Nov 02 15:56:51.331208 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:51.331177    7019 scope.go:122] "RemoveContainer" containerID="64a8d5dfcd04413bf1cfd2cb85883f72f3faa61c29ea568df502823f8a5f2750"
Nov 02 15:56:51.341027 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:51.341003    7019 scope.go:122] "RemoveContainer" containerID="0dedef7b3f2ec9205f8cc64668e04f4d3c5fbe79fe9d8d17ed785c5514d14f2e"
Nov 02 15:56:52.704499 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:56:52.704458    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:56:52.808143 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:52.808113    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:52.808590 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:52.808564    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:52.809296 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:52.809269    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-8l7bs"
Nov 02 15:56:56.470489 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:56:56.470444    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:57:10.063838 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:10.063771    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:57:11.362637 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:11.362602    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:57:13.453995 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:13.453959    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:57:15.131956 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:15.131921    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:57:22.704151 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:57:22.704106    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:57:29.075000 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.074960    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"]
Nov 02 15:57:29.078739 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.078721    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.110167 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.110133    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"]
Nov 02 15:57:29.147680 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147626    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbzkv\" (UniqueName: \"kubernetes.io/projected/30ae8e5d-46c2-4af1-9ea4-e27446758848-kube-api-access-tbzkv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.147872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147733    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.147872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147771    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.147872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147820    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.147872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147839    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.147872 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147860    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.148192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147878    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.148192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147937    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.148192 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.147972    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae8e5d-46c2-4af1-9ea4-e27446758848-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249071 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249033    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249089    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249110    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249129    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249146    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249280 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249186    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249333    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae8e5d-46c2-4af1-9ea4-e27446758848-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249398    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbzkv\" (UniqueName: \"kubernetes.io/projected/30ae8e5d-46c2-4af1-9ea4-e27446758848-kube-api-access-tbzkv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249436    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249611 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249571    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-workload-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249613    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-workload-certs\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.249840 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249793    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-credential-socket\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.250012 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.249994    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-data\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.250077 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.250061    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ae8e5d-46c2-4af1-9ea4-e27446758848-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.252110 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.252078    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-envoy\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.252321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.252302    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.273386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.273364    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ae8e5d-46c2-4af1-9ea4-e27446758848-istio-token\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.273386 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.273372    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbzkv\" (UniqueName: \"kubernetes.io/projected/30ae8e5d-46c2-4af1-9ea4-e27446758848-kube-api-access-tbzkv\") pod \"maas-default-gateway-openshift-default-fcb75c45f-cq5h5\" (UID: \"30ae8e5d-46c2-4af1-9ea4-e27446758848\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.392103 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.392071    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:29.718913 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.718884    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"]
Nov 02 15:57:29.719709 ip-10-0-129-45 kubenswrapper[7019]: W1102 15:57:29.719684    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ae8e5d_46c2_4af1_9ea4_e27446758848.slice/crio-1837a3613979a1f77578ef3ebc37efa90984c5278e4251f1a2494d0db0a4cbdd WatchSource:0}: Error finding container 1837a3613979a1f77578ef3ebc37efa90984c5278e4251f1a2494d0db0a4cbdd: Status 404 returned error can't find the container with id 1837a3613979a1f77578ef3ebc37efa90984c5278e4251f1a2494d0db0a4cbdd
Nov 02 15:57:29.721805 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.721769    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:29.721906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.721838    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:29.721906 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:29.721865    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 15:57:30.398327 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:30.398284    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" event={"ID":"30ae8e5d-46c2-4af1-9ea4-e27446758848","Type":"ContainerStarted","Data":"91f8cc10d64890a04db82ded9e75091e231fd4262b64636f89c1a6040ba7d735"}
Nov 02 15:57:30.398757 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:30.398338    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" event={"ID":"30ae8e5d-46c2-4af1-9ea4-e27446758848","Type":"ContainerStarted","Data":"1837a3613979a1f77578ef3ebc37efa90984c5278e4251f1a2494d0db0a4cbdd"}
Nov 02 15:57:30.446823 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:30.446762    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" podStartSLOduration=1.446749019 podStartE2EDuration="1.446749019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:57:30.445249666 +0000 UTC m=+3644.699203291" watchObservedRunningTime="2026-11-02 15:57:30.446749019 +0000 UTC m=+3644.700702643"
Nov 02 15:57:31.392734 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:31.392698    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:31.394339 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:31.394306    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cq5h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body=
Nov 02 15:57:31.394492 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:31.394373    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" podUID="30ae8e5d-46c2-4af1-9ea4-e27446758848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused"
Nov 02 15:57:32.393392 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:32.393350    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cq5h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body=
Nov 02 15:57:32.393950 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:32.393430    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" podUID="30ae8e5d-46c2-4af1-9ea4-e27446758848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused"
Nov 02 15:57:33.392667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:33.392607    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cq5h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body=
Nov 02 15:57:33.392882 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:33.392679    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" podUID="30ae8e5d-46c2-4af1-9ea4-e27446758848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused"
Nov 02 15:57:34.393876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:34.393383    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cq5h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body=
Nov 02 15:57:34.393876 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:34.393444    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" podUID="30ae8e5d-46c2-4af1-9ea4-e27446758848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused"
Nov 02 15:57:35.393321 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:35.393283    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cq5h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body=
Nov 02 15:57:35.393528 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:35.393341    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" podUID="30ae8e5d-46c2-4af1-9ea4-e27446758848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused"
Nov 02 15:57:36.392501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:36.392451    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-cq5h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body=
Nov 02 15:57:36.393074 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:36.392540    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5" podUID="30ae8e5d-46c2-4af1-9ea4-e27446758848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused"
Nov 02 15:57:37.873369 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:37.873329    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:37.873843 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:37.873615    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:37.874868 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:37.874845    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-cq5h5"
Nov 02 15:57:42.029737 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:42.029699    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"]
Nov 02 15:57:42.042293 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:42.042263    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ad377c3-pc64k"]
Nov 02 15:57:43.860686 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:43.860636    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8981d766-11b7-48e8-8341-26589b76aab1" path="/var/lib/kubelet/pods/8981d766-11b7-48e8-8341-26589b76aab1/volumes"
Nov 02 15:57:50.031302 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:50.031270    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"]
Nov 02 15:57:50.043973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:50.043941    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"]
Nov 02 15:57:50.045320 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:50.045309    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbc7a634-rdxgd"]
Nov 02 15:57:50.046628 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:50.046609    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98212ddf-bbbzc"]
Nov 02 15:57:51.378419 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.378381    7019 scope.go:122] "RemoveContainer" containerID="0572a23feb50afca167fbf989f9224eb107fabd01aad48abc89df556ce34f1fd"
Nov 02 15:57:51.388155 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.388130    7019 scope.go:122] "RemoveContainer" containerID="7c0284ac9cb9ecebc5348fb0d816f1646c63fb75b3bfa878119d1a5c7ebe86f0"
Nov 02 15:57:51.397088 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.397068    7019 scope.go:122] "RemoveContainer" containerID="d9376ce1f480cf34152700a51c359bc8543fd67a0cf5b66e916988409c745f12"
Nov 02 15:57:51.405148 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.405127    7019 scope.go:122] "RemoveContainer" containerID="d5344835370283329dbac958766144c0423bab884fd39ebf25c5d84e9d9b315d"
Nov 02 15:57:51.413234 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.413217    7019 scope.go:122] "RemoveContainer" containerID="9d4efd8d90f476978d3e54d53a6aa279365653cca8b0d7ff19be642f57c5bea2"
Nov 02 15:57:51.421704 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.421680    7019 scope.go:122] "RemoveContainer" containerID="4e2739f10446d2ce20de81ab88ee6136f2c5f149d2ca9c1b677a3c0d55d86898"
Nov 02 15:57:51.860916 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.860890    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b52288-5405-47fc-9f32-7fa34431d14e" path="/var/lib/kubelet/pods/c0b52288-5405-47fc-9f32-7fa34431d14e/volumes"
Nov 02 15:57:51.861322 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:51.861309    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f665ad06-d721-4d30-ba4d-7ad32c9638f1" path="/var/lib/kubelet/pods/f665ad06-d721-4d30-ba4d-7ad32c9638f1/volumes"
Nov 02 15:57:52.705109 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:57:52.705065    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:57:53.371010 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:53.370977    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:57:56.651893 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:57:56.651841    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:10.162087 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:10.162053    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:10.549923 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:10.549834    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:58:15.183911 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:15.183870    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:58:15.253389 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:15.253350    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:17.826670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:17.826352    7019 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fcb75c45f-xqbf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:58:17.826670 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:17.826433    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fcb75c45f-xqbf4" podUID="d1affeef-d790-45c6-b74c-9053b4f08f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:58:19.023198 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:19.023163    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:21.875742 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:21.875702    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:22.707747 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:58:22.707709    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:58:30.564530 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:30.564440    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:58:46.475814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:46.475775    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:58:48.604669 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:48.604158    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:48.746324 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:48.746288    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:51.466441 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.466397    7019 scope.go:122] "RemoveContainer" containerID="32a0ad1ca01ce000db95e452a90f59b1bdd20b5d50a4a0a260d638c3b43a8c6b"
Nov 02 15:58:51.475667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.475635    7019 scope.go:122] "RemoveContainer" containerID="5dfc24c0ab440357aa032fa7cf253daca660e81c5335b5f10d1e5d485ddf3a1d"
Nov 02 15:58:51.484312 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.484290    7019 scope.go:122] "RemoveContainer" containerID="255f06216255fc0af29ab3469f5b7230329dc13e28e81e63215dce9ce836d256"
Nov 02 15:58:51.492936 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.492917    7019 scope.go:122] "RemoveContainer" containerID="d80d845f5b51ce397d24268765180437843c235fe1e9374e780221baafe9493b"
Nov 02 15:58:51.500837 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.500815    7019 scope.go:122] "RemoveContainer" containerID="31b35ceb6846f1f52d1c10a0e32c24fd8332a9fac700d5e522c66eff5f6f6c83"
Nov 02 15:58:51.508505 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.508485    7019 scope.go:122] "RemoveContainer" containerID="d8ddea6226bad38c80dcbf20a810a283ba0659c1660c07b8e70da5a2086b6630"
Nov 02 15:58:51.518874 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.518852    7019 scope.go:122] "RemoveContainer" containerID="97b811f9f8c0902f1cfee71e3b94c2d57c6e0d4a271845cac681a5d278c41258"
Nov 02 15:58:51.527053 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.527034    7019 scope.go:122] "RemoveContainer" containerID="34ecbfd5c5203abd634ba8a3ff97f6276e51893c2a24741626c1dec972cdead3"
Nov 02 15:58:51.534782 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.534765    7019 scope.go:122] "RemoveContainer" containerID="7fbab9c76f871b07f869cf33006385f0e3640400549c3351a38f3e2d6cdbd2ea"
Nov 02 15:58:51.542690 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:51.542672    7019 scope.go:122] "RemoveContainer" containerID="fd246a0fed08a09b8b596c50c3ccc29f52e5be959f9f4ece4d5c8e9fd9d96de0"
Nov 02 15:58:52.704949 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:58:52.704910    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:58:55.405973 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:55.405937    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:55.551581 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:55.551545    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:58:56.761773 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:58:56.761733    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:59:02.032182 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:02.032140    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"]
Nov 02 15:59:02.044850 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:02.044816    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad7088e-nn74x"]
Nov 02 15:59:02.046174 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:02.046159    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"]
Nov 02 15:59:02.047478 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:02.047455    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9021546-sd4gd"]
Nov 02 15:59:03.860667 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:03.860617    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e50a1ed-55c8-488f-b324-fa12bfe3bb7f" path="/var/lib/kubelet/pods/0e50a1ed-55c8-488f-b324-fa12bfe3bb7f/volumes"
Nov 02 15:59:03.861079 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:03.861063    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8eabab3-0500-4f63-9d5f-4b6aa5097970" path="/var/lib/kubelet/pods/a8eabab3-0500-4f63-9d5f-4b6aa5097970/volumes"
Nov 02 15:59:05.233743 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:05.233705    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:59:10.299184 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:10.299143    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:59:10.334015 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:10.333978    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:59:15.363801 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:15.363759    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:59:20.731242 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:20.731202    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:59:22.706763 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:59:22.706720    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:59:25.317575 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:25.317541    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:59:28.037862 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:28.037820    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"]
Nov 02 15:59:28.049570 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:28.049535    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"]
Nov 02 15:59:28.054104 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:28.054074    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0223a1d9-xwd5r"]
Nov 02 15:59:28.056730 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:28.056704    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b5d512-p6s6v"]
Nov 02 15:59:29.860753 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:29.860720    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908b243c-48a9-4485-a338-6cc361f65ad8" path="/var/lib/kubelet/pods/908b243c-48a9-4485-a338-6cc361f65ad8/volumes"
Nov 02 15:59:29.861243 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:29.861128    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf9cf03-3a92-418d-a9b9-72caba0ad432" path="/var/lib/kubelet/pods/bdf9cf03-3a92-418d-a9b9-72caba0ad432/volumes"
Nov 02 15:59:45.007634 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:45.007596    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 15:59:47.156501 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:47.156470    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:59:51.604702 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:51.604382    7019 scope.go:122] "RemoveContainer" containerID="7b961fc329b35ce7fac3fb0027dd9d5ddc0c2e4c580aa19a6ca66f503cb4b2e4"
Nov 02 15:59:51.647814 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:51.647784    7019 scope.go:122] "RemoveContainer" containerID="23522528467dba82be2ae8b5d4878eb5ace88c9f7df2f41f1b6495d04db8f7c1"
Nov 02 15:59:51.658624 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:51.658596    7019 scope.go:122] "RemoveContainer" containerID="3e436749bdb198f015343782fbd23dc643334eac0f55a14cec27ffd9e8a31d0e"
Nov 02 15:59:51.716699 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:51.714798    7019 scope.go:122] "RemoveContainer" containerID="661f39b386c8df807b972580d2728c2f60a6e72556ad209b9e81332a626887f5"
Nov 02 15:59:52.516556 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:52.510495    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 15:59:52.705669 ip-10-0-129-45 kubenswrapper[7019]: E1102 15:59:52.705436    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 15:59:56.883988 ip-10-0-129-45 kubenswrapper[7019]: I1102 15:59:56.883903    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:00:00.141415 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.141374    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"]
Nov 02 16:00:00.147551 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.147523    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.164454 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.164415    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"]
Nov 02 16:00:00.175832 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.175810    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:00:00.238083 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.238051    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-2ztrs\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.238269 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.238095    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvr4\" (UniqueName: \"kubernetes.io/projected/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-kube-api-access-lxvr4\") pod \"maas-api-key-cleanup-29893920-2ztrs\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.339052 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.339016    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-2ztrs\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.339230 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.339060    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvr4\" (UniqueName: \"kubernetes.io/projected/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-kube-api-access-lxvr4\") pod \"maas-api-key-cleanup-29893920-2ztrs\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.339701 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.339679    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893920-2ztrs\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.364101 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.364064    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvr4\" (UniqueName: \"kubernetes.io/projected/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-kube-api-access-lxvr4\") pod \"maas-api-key-cleanup-29893920-2ztrs\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.461279 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.461191    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:00.759897 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:00:00.759866    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeb7f34_a7d3_4a52_971a_6cd5c5051123.slice/crio-d95813e9cd29ea5e3a06492eba18cf0b7bf9bf82a3218fb256e851da9ab823ec WatchSource:0}: Error finding container d95813e9cd29ea5e3a06492eba18cf0b7bf9bf82a3218fb256e851da9ab823ec: Status 404 returned error can't find the container with id d95813e9cd29ea5e3a06492eba18cf0b7bf9bf82a3218fb256e851da9ab823ec
Nov 02 16:00:00.760076 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:00.760051    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"]
Nov 02 16:00:01.378604 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:01.378568    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" event={"ID":"dbeb7f34-a7d3-4a52-971a-6cd5c5051123","Type":"ContainerStarted","Data":"38c864523dace7b781e78dedd456e21ec70a4a2d6ab7c19fb7ecb25cd2bc245b"}
Nov 02 16:00:01.379026 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:01.378626    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" event={"ID":"dbeb7f34-a7d3-4a52-971a-6cd5c5051123","Type":"ContainerStarted","Data":"d95813e9cd29ea5e3a06492eba18cf0b7bf9bf82a3218fb256e851da9ab823ec"}
Nov 02 16:00:01.421219 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:01.421158    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" podStartSLOduration=1.421145365 podStartE2EDuration="1.421145365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:00:01.420398621 +0000 UTC m=+3795.674352246" watchObservedRunningTime="2026-11-02 16:00:01.421145365 +0000 UTC m=+3795.675098989"
Nov 02 16:00:02.385415 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:02.385382    7019 generic.go:356] "Generic (PLEG): container finished" podID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerID="38c864523dace7b781e78dedd456e21ec70a4a2d6ab7c19fb7ecb25cd2bc245b" exitCode=60
Nov 02 16:00:02.385873 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:02.385447    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" event={"ID":"dbeb7f34-a7d3-4a52-971a-6cd5c5051123","Type":"ContainerDied","Data":"38c864523dace7b781e78dedd456e21ec70a4a2d6ab7c19fb7ecb25cd2bc245b"}
Nov 02 16:00:02.385873 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:02.385714    7019 scope.go:122] "RemoveContainer" containerID="38c864523dace7b781e78dedd456e21ec70a4a2d6ab7c19fb7ecb25cd2bc245b"
Nov 02 16:00:03.392717 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:03.392677    7019 generic.go:356] "Generic (PLEG): container finished" podID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerID="e9ef3fe66f362f48d2b210871e6cbf5e2ba065f8280c99de606263a18f99e602" exitCode=60
Nov 02 16:00:03.393147 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:03.392763    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" event={"ID":"dbeb7f34-a7d3-4a52-971a-6cd5c5051123","Type":"ContainerDied","Data":"e9ef3fe66f362f48d2b210871e6cbf5e2ba065f8280c99de606263a18f99e602"}
Nov 02 16:00:03.393147 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:03.392825    7019 scope.go:122] "RemoveContainer" containerID="38c864523dace7b781e78dedd456e21ec70a4a2d6ab7c19fb7ecb25cd2bc245b"
Nov 02 16:00:03.393147 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:03.393070    7019 scope.go:122] "RemoveContainer" containerID="e9ef3fe66f362f48d2b210871e6cbf5e2ba065f8280c99de606263a18f99e602"
Nov 02 16:00:03.393340 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:00:03.393320    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-2ztrs_odh-ai-gateway-infra(dbeb7f34-a7d3-4a52-971a-6cd5c5051123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123"
Nov 02 16:00:04.399690 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:04.399664    7019 scope.go:122] "RemoveContainer" containerID="e9ef3fe66f362f48d2b210871e6cbf5e2ba065f8280c99de606263a18f99e602"
Nov 02 16:00:04.400058 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:00:04.399896    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-2ztrs_odh-ai-gateway-infra(dbeb7f34-a7d3-4a52-971a-6cd5c5051123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123"
Nov 02 16:00:04.628019 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:04.627985    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:00:05.588748 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:05.588710    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:00:09.061058 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:09.061019    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:00:10.418111 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:10.418072    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:00:13.617686 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:13.617636    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:00:13.854697 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:13.854634    7019 scope.go:122] "RemoveContainer" containerID="e9ef3fe66f362f48d2b210871e6cbf5e2ba065f8280c99de606263a18f99e602"
Nov 02 16:00:14.461882 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:14.461789    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" event={"ID":"dbeb7f34-a7d3-4a52-971a-6cd5c5051123","Type":"ContainerStarted","Data":"0cb3df2406e08a74897543b0017a7b8694c1dbb9463dbabbabba7c5f6e47502f"}
Nov 02 16:00:15.475209 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:15.475085    7019 generic.go:356] "Generic (PLEG): container finished" podID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerID="0cb3df2406e08a74897543b0017a7b8694c1dbb9463dbabbabba7c5f6e47502f" exitCode=60
Nov 02 16:00:15.476026 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:15.475924    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" event={"ID":"dbeb7f34-a7d3-4a52-971a-6cd5c5051123","Type":"ContainerDied","Data":"0cb3df2406e08a74897543b0017a7b8694c1dbb9463dbabbabba7c5f6e47502f"}
Nov 02 16:00:15.476228 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:15.476158    7019 scope.go:122] "RemoveContainer" containerID="e9ef3fe66f362f48d2b210871e6cbf5e2ba065f8280c99de606263a18f99e602"
Nov 02 16:00:15.476977 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:15.476875    7019 scope.go:122] "RemoveContainer" containerID="0cb3df2406e08a74897543b0017a7b8694c1dbb9463dbabbabba7c5f6e47502f"
Nov 02 16:00:15.477503 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:00:15.477404    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893920-2ztrs_odh-ai-gateway-infra(dbeb7f34-a7d3-4a52-971a-6cd5c5051123)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123"
Nov 02 16:00:15.518346 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:15.517557    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"]
Nov 02 16:00:15.584286 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:15.584181    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:00:16.795414 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:16.795387    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:16.865806 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:16.865771    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-kube-api-access-lxvr4\" (UniqueName: \"kubernetes.io/projected/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-kube-api-access-lxvr4\") pod \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") "
Nov 02 16:00:16.865976 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:16.865848    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-maas-api-service-ca\") pod \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\" (UID: \"dbeb7f34-a7d3-4a52-971a-6cd5c5051123\") "
Nov 02 16:00:16.866200 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:16.866177    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-maas-api-service-ca" pod "dbeb7f34-a7d3-4a52-971a-6cd5c5051123" (UID: "dbeb7f34-a7d3-4a52-971a-6cd5c5051123"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:00:16.868096 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:16.868071    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-kube-api-access-lxvr4" pod "dbeb7f34-a7d3-4a52-971a-6cd5c5051123" (UID: "dbeb7f34-a7d3-4a52-971a-6cd5c5051123"). InnerVolumeSpecName "kube-api-access-lxvr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:00:16.967312 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:16.967275    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvr4\" (UniqueName: \"kubernetes.io/projected/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-kube-api-access-lxvr4\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:00:16.967312 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:16.967308    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbeb7f34-a7d3-4a52-971a-6cd5c5051123-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:00:17.511036 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:17.511006    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"
Nov 02 16:00:17.511217 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:17.511010    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs" event={"ID":"dbeb7f34-a7d3-4a52-971a-6cd5c5051123","Type":"ContainerDied","Data":"d95813e9cd29ea5e3a06492eba18cf0b7bf9bf82a3218fb256e851da9ab823ec"}
Nov 02 16:00:17.511217 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:17.511135    7019 scope.go:122] "RemoveContainer" containerID="0cb3df2406e08a74897543b0017a7b8694c1dbb9463dbabbabba7c5f6e47502f"
Nov 02 16:00:17.565018 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:17.564987    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"]
Nov 02 16:00:17.569060 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:17.569032    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893920-2ztrs"]
Nov 02 16:00:17.860725 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:17.860700    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" path="/var/lib/kubelet/pods/dbeb7f34-a7d3-4a52-971a-6cd5c5051123/volumes"
Nov 02 16:00:22.704902 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:00:22.704867    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:00:29.244723 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:29.244690    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:00:32.038034 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:32.037997    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"]
Nov 02 16:00:32.050715 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:32.050673    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c911da5-sqvzq"]
Nov 02 16:00:33.862638 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:33.862603    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2" path="/var/lib/kubelet/pods/3f38c348-7d9b-4e3c-ba2f-b4d6d75ba0e2/volumes"
Nov 02 16:00:34.380674 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:34.372174    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:00:50.451431 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:50.451397    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:00:51.824770 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:51.824737    7019 scope.go:122] "RemoveContainer" containerID="59058f7075d00028bb6e1f51037a132c03007423d592f13594c64fea051e8294"
Nov 02 16:00:51.833633 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:51.833605    7019 scope.go:122] "RemoveContainer" containerID="4e075bbc389f69455bc9a13f28113d13161c0a562046e43ecb5a78bfc9e4e8b4"
Nov 02 16:00:51.842270 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:51.842245    7019 scope.go:122] "RemoveContainer" containerID="9e135ce84146743d906eda7757e91ba4524cdca3871e802f2f500acca3c8ce4c"
Nov 02 16:00:51.852128 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:51.852107    7019 scope.go:122] "RemoveContainer" containerID="d7548483b7bf5926523e00dc740aa8dc6d690a283db2b23908bd74688ba67308"
Nov 02 16:00:51.861713 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:51.861694    7019 scope.go:122] "RemoveContainer" containerID="13b2d11034ea0fc162fd4b1743b4ecee4489b7d229d76663cdff65e75b3b9ffe"
Nov 02 16:00:51.869884 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:51.869866    7019 scope.go:122] "RemoveContainer" containerID="0566a03b9369f5ff4739530f7e1f3360f254aa3c3d481c88de79971a9da6c932"
Nov 02 16:00:52.704734 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:00:52.704699    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:00:57.028905 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:00:57.028868    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:01:01.732576 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:01.732541    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:01:07.577176 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:07.577139    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:01:20.527477 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:20.527443    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:01:22.705100 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:01:22.705053    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:01:25.121792 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:25.121757    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:01:25.690338 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:25.690295    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:01:30.315384 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:30.315278    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:01:31.507107 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:31.507075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:01:33.814553 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:33.814518    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:01:40.067331 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:40.067294    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:01:51.919765 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:51.919730    7019 scope.go:122] "RemoveContainer" containerID="24ae1afef37f12921f805778eefbff0260e2033d0042443ca76a66904b886039"
Nov 02 16:01:51.928217 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:51.928196    7019 scope.go:122] "RemoveContainer" containerID="920a0d4e4b14bfe433aec5634da25cb613fba7530db656de56cb78489d5a305e"
Nov 02 16:01:52.704116 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:01:52.704070    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:01:57.168731 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:01:57.168697    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:00.628690 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:00.628627    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:02:11.024597 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:11.024553    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:02:20.709338 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:20.709307    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:22.703635 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:02:22.703600    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:02:25.821197 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:25.821159    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:33.905715 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:33.905679    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:34.394834 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:34.394777    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:34.547421 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:34.547378    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:38.249795 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:38.249756    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:38.401740 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:38.401704    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:39.811178 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:39.811144    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:39.973551 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:39.973508    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:40.147704 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:40.147643    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:41.614270 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:41.614234    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:43.310308 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:43.310268    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:43.666064 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:43.666015    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:02:45.049975 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:45.049938    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:46.930882 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:46.930844    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:48.777883 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:48.777845    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:50.612803 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:50.612772    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:52.324506 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:52.324460    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:52.703895 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:02:52.703849    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:02:53.023582 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:53.023486    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:53.621566 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:53.621531    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:53.777212 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:53.777177    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:53.941421 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:53.941321    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:54.202367 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:54.202275    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:55.718691 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:55.718629    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:56.039257 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:56.039163    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:57.939532 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:57.939448    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:02:57.960379 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:57.960331    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:02:59.820440 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:02:59.820402    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:01.611660 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:01.611613    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:03.309724 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:03.309681    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:05.004155 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:05.004119    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:06.810829 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:06.810789    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:08.600751 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:08.600715    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:10.298222 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:10.298182    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:12.308229 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:12.308190    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:13.518038 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:13.518001    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:03:14.020756 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:14.020721    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:15.805626 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:15.805588    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:17.600859 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:17.600819    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:19.416300 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:19.416260    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:20.807814 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:20.807776    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:21.108135 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:21.108047    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:22.704796 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:03:22.704764    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:03:22.812709 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:22.812677    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:24.516845 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:24.516809    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:25.950106 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:25.950072    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:26.216670 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:26.216574    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:28.022822 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:28.022778    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:29.715984 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:29.715945    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:31.509083 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:31.509049    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:33.218851 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:33.218813    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:35.006328 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:35.006287    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:36.796805 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:36.796769    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:37.218144 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218105    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"]
Nov 02 16:03:37.218605 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218584    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.218723 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218606    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.218723 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218670    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.218723 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218679    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.218723 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218693    7019 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.218723 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218700    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.218994 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218818    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.218994 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.218839    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:37.221758 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.221739    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.239637 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.239615    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:03:37.252438 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.252413    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:03:37.253608 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.253588    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"]
Nov 02 16:03:37.280991 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.280957    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.281131 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.281029    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.281205 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.281182    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xxj\" (UniqueName: \"kubernetes.io/projected/a284242a-b7f7-4ef3-8626-28301d724b4e-kube-api-access-z4xxj\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.382671 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.382610    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.382830 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.382692    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.382830 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.382806    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xxj\" (UniqueName: \"kubernetes.io/projected/a284242a-b7f7-4ef3-8626-28301d724b4e-kube-api-access-z4xxj\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.383445 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.383423    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.385049 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.385030    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.407202 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.407177    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xxj\" (UniqueName: \"kubernetes.io/projected/a284242a-b7f7-4ef3-8626-28301d724b4e-kube-api-access-z4xxj\") pod \"maas-api-revoke-sub-d25ac2ab-qz2j2\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.534926 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.534839    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:37.833932 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:03:37.833901    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda284242a_b7f7_4ef3_8626_28301d724b4e.slice/crio-b9cc6ec58fd4f0e797bf680137092ee8bde085928913ee558315c33de3c8934e WatchSource:0}: Error finding container b9cc6ec58fd4f0e797bf680137092ee8bde085928913ee558315c33de3c8934e: Status 404 returned error can't find the container with id b9cc6ec58fd4f0e797bf680137092ee8bde085928913ee558315c33de3c8934e
Nov 02 16:03:37.834966 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:37.834934    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"]
Nov 02 16:03:38.609014 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:38.608934    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:38.649257 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:38.648172    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2" event={"ID":"a284242a-b7f7-4ef3-8626-28301d724b4e","Type":"ContainerStarted","Data":"e599cfbe208f7537ac2fa01d4e812add724b0b1848c9e95312aef4c9d1120d5c"}
Nov 02 16:03:38.649257 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:38.648297    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2" event={"ID":"a284242a-b7f7-4ef3-8626-28301d724b4e","Type":"ContainerStarted","Data":"b9cc6ec58fd4f0e797bf680137092ee8bde085928913ee558315c33de3c8934e"}
Nov 02 16:03:38.699907 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:38.699860    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2" podStartSLOduration=1.699847109 podStartE2EDuration="1.699847109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:03:38.699050695 +0000 UTC m=+4012.953004320" watchObservedRunningTime="2026-11-02 16:03:38.699847109 +0000 UTC m=+4012.953800733"
Nov 02 16:03:39.654588 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:39.654554    7019 generic.go:356] "Generic (PLEG): container finished" podID="a284242a-b7f7-4ef3-8626-28301d724b4e" containerID="e599cfbe208f7537ac2fa01d4e812add724b0b1848c9e95312aef4c9d1120d5c" exitCode=0
Nov 02 16:03:39.655021 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:39.654655    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2" event={"ID":"a284242a-b7f7-4ef3-8626-28301d724b4e","Type":"ContainerDied","Data":"e599cfbe208f7537ac2fa01d4e812add724b0b1848c9e95312aef4c9d1120d5c"}
Nov 02 16:03:40.325697 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.325634    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:40.795389 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.795364    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:40.916626 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.916540    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-openshift-ca\") pod \"a284242a-b7f7-4ef3-8626-28301d724b4e\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") "
Nov 02 16:03:40.916626 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.916600    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-xks-ca\") pod \"a284242a-b7f7-4ef3-8626-28301d724b4e\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") "
Nov 02 16:03:40.916921 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.916755    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a284242a-b7f7-4ef3-8626-28301d724b4e-kube-api-access-z4xxj\" (UniqueName: \"kubernetes.io/projected/a284242a-b7f7-4ef3-8626-28301d724b4e-kube-api-access-z4xxj\") pod \"a284242a-b7f7-4ef3-8626-28301d724b4e\" (UID: \"a284242a-b7f7-4ef3-8626-28301d724b4e\") "
Nov 02 16:03:40.916984 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.916960    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-openshift-ca" pod "a284242a-b7f7-4ef3-8626-28301d724b4e" (UID: "a284242a-b7f7-4ef3-8626-28301d724b4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:03:40.918696 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.918657    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-xks-ca" pod "a284242a-b7f7-4ef3-8626-28301d724b4e" (UID: "a284242a-b7f7-4ef3-8626-28301d724b4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:03:40.918824 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:40.918730    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a284242a-b7f7-4ef3-8626-28301d724b4e-kube-api-access-z4xxj" pod "a284242a-b7f7-4ef3-8626-28301d724b4e" (UID: "a284242a-b7f7-4ef3-8626-28301d724b4e"). InnerVolumeSpecName "kube-api-access-z4xxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:03:41.018444 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:41.018411    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4xxj\" (UniqueName: \"kubernetes.io/projected/a284242a-b7f7-4ef3-8626-28301d724b4e-kube-api-access-z4xxj\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:03:41.018444 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:41.018441    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:03:41.018680 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:41.018456    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a284242a-b7f7-4ef3-8626-28301d724b4e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:03:41.665271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:41.665238    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"
Nov 02 16:03:41.665271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:41.665262    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2" event={"ID":"a284242a-b7f7-4ef3-8626-28301d724b4e","Type":"ContainerDied","Data":"b9cc6ec58fd4f0e797bf680137092ee8bde085928913ee558315c33de3c8934e"}
Nov 02 16:03:41.665504 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:41.665314    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9cc6ec58fd4f0e797bf680137092ee8bde085928913ee558315c33de3c8934e"
Nov 02 16:03:42.115555 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:42.115523    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:43.805370 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:43.805324    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:45.607265 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:45.607230    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:47.399374 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:47.399338    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:49.195897 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:49.195859    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:49.827609 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:49.827571    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:03:51.033264 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:51.033226    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:52.705924 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:03:52.705876    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:03:52.904323 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:52.904288    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:54.696491 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:54.696434    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:56.403869 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:56.403835    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:58.103214 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.103179    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:58.235032 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.234991    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:03:58.439924 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.439887    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"]
Nov 02 16:03:58.440485 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.440462    7019 cpu_manager.go:414] "containerMap: removing container" podUID="a284242a-b7f7-4ef3-8626-28301d724b4e" containerName="revoke-keys"
Nov 02 16:03:58.440485 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.440482    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a284242a-b7f7-4ef3-8626-28301d724b4e" containerName="revoke-keys"
Nov 02 16:03:58.440642 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.440566    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb7f34-a7d3-4a52-971a-6cd5c5051123" containerName="cleanup"
Nov 02 16:03:58.440642 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.440575    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="a284242a-b7f7-4ef3-8626-28301d724b4e" containerName="revoke-keys"
Nov 02 16:03:58.443565 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.443547    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.474491 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.474460    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:03:58.474633 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.474497    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:03:58.476121 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.476100    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"]
Nov 02 16:03:58.502395 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.502363    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9486r\" (UniqueName: \"kubernetes.io/projected/ed41b7fd-f044-40c0-a392-8da7588dc5f1-kube-api-access-9486r\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.502538 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.502404    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.502538 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.502516    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.603022 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.602984    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.603187 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.603080    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9486r\" (UniqueName: \"kubernetes.io/projected/ed41b7fd-f044-40c0-a392-8da7588dc5f1-kube-api-access-9486r\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.603187 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.603108    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.603628 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.603605    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.605447 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.605424    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.627394 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.627368    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9486r\" (UniqueName: \"kubernetes.io/projected/ed41b7fd-f044-40c0-a392-8da7588dc5f1-kube-api-access-9486r\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:58.757291 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:58.757215    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:03:59.054367 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:59.054335    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"]
Nov 02 16:03:59.055543 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:03:59.055518    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded41b7fd_f044_40c0_a392_8da7588dc5f1.slice/crio-ab08275ec1be3a1a21f50e3a09a6910aa4476863061bac9c302fb393cdea46a1 WatchSource:0}: Error finding container ab08275ec1be3a1a21f50e3a09a6910aa4476863061bac9c302fb393cdea46a1: Status 404 returned error can't find the container with id ab08275ec1be3a1a21f50e3a09a6910aa4476863061bac9c302fb393cdea46a1
Nov 02 16:03:59.758503 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:59.758411    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77" event={"ID":"ed41b7fd-f044-40c0-a392-8da7588dc5f1","Type":"ContainerStarted","Data":"d0005a5c45d60c157a08aa0334958576e69aefcfad7ae9c37f8978012484ad77"}
Nov 02 16:03:59.758503 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:59.758472    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77" event={"ID":"ed41b7fd-f044-40c0-a392-8da7588dc5f1","Type":"ContainerStarted","Data":"ab08275ec1be3a1a21f50e3a09a6910aa4476863061bac9c302fb393cdea46a1"}
Nov 02 16:03:59.802261 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:59.802209    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77" podStartSLOduration=1.802195491 podStartE2EDuration="1.802195491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:03:59.800524913 +0000 UTC m=+4034.054478539" watchObservedRunningTime="2026-11-02 16:03:59.802195491 +0000 UTC m=+4034.056149128"
Nov 02 16:03:59.918888 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:03:59.918857    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:00.764708 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:00.764637    7019 generic.go:356] "Generic (PLEG): container finished" podID="ed41b7fd-f044-40c0-a392-8da7588dc5f1" containerID="d0005a5c45d60c157a08aa0334958576e69aefcfad7ae9c37f8978012484ad77" exitCode=0
Nov 02 16:04:00.765151 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:00.764677    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77" event={"ID":"ed41b7fd-f044-40c0-a392-8da7588dc5f1","Type":"ContainerDied","Data":"d0005a5c45d60c157a08aa0334958576e69aefcfad7ae9c37f8978012484ad77"}
Nov 02 16:04:01.633712 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.633631    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:01.914912 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.914879    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:04:01.933884 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.933849    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed41b7fd-f044-40c0-a392-8da7588dc5f1-kube-api-access-9486r\" (UniqueName: \"kubernetes.io/projected/ed41b7fd-f044-40c0-a392-8da7588dc5f1-kube-api-access-9486r\") pod \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") "
Nov 02 16:04:01.933884 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.933894    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-xks-ca\") pod \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") "
Nov 02 16:04:01.934130 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.933925    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-openshift-ca\") pod \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\" (UID: \"ed41b7fd-f044-40c0-a392-8da7588dc5f1\") "
Nov 02 16:04:01.934335 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.934302    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-openshift-ca" pod "ed41b7fd-f044-40c0-a392-8da7588dc5f1" (UID: "ed41b7fd-f044-40c0-a392-8da7588dc5f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:04:01.936216 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.936180    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-xks-ca" pod "ed41b7fd-f044-40c0-a392-8da7588dc5f1" (UID: "ed41b7fd-f044-40c0-a392-8da7588dc5f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:04:01.936329 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:01.936235    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed41b7fd-f044-40c0-a392-8da7588dc5f1-kube-api-access-9486r" pod "ed41b7fd-f044-40c0-a392-8da7588dc5f1" (UID: "ed41b7fd-f044-40c0-a392-8da7588dc5f1"). InnerVolumeSpecName "kube-api-access-9486r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:02.034797 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:02.034760    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9486r\" (UniqueName: \"kubernetes.io/projected/ed41b7fd-f044-40c0-a392-8da7588dc5f1-kube-api-access-9486r\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:04:02.034797 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:02.034795    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:04:02.034797 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:02.034805    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41b7fd-f044-40c0-a392-8da7588dc5f1-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:04:02.271866 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:02.271768    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:02.776140 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:02.776096    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77" event={"ID":"ed41b7fd-f044-40c0-a392-8da7588dc5f1","Type":"ContainerDied","Data":"ab08275ec1be3a1a21f50e3a09a6910aa4476863061bac9c302fb393cdea46a1"}
Nov 02 16:04:02.776140 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:02.776135    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"
Nov 02 16:04:02.776368 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:02.776160    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab08275ec1be3a1a21f50e3a09a6910aa4476863061bac9c302fb393cdea46a1"
Nov 02 16:04:03.257936 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:03.257893    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:03.393042 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:03.393004    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:05.027830 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:05.027779    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:06.791421 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:06.791378    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:08.500831 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:08.500788    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:09.089460 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:09.089425    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:09.229724 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:09.229686    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:10.221876 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:10.221838    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:11.694460 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:11.694418    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:12.264892 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:12.264860    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:12.411895 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:12.411855    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"]
Nov 02 16:04:12.417634 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:12.417606    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:12.424209 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:12.424186    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fee58f-c5n77"]
Nov 02 16:04:12.937174 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:12.937137    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:13.860524 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:13.860490    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed41b7fd-f044-40c0-a392-8da7588dc5f1" path="/var/lib/kubelet/pods/ed41b7fd-f044-40c0-a392-8da7588dc5f1/volumes"
Nov 02 16:04:14.142668 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:14.142578    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:04:19.054833 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:19.054782    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:19.195758 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:19.195689    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:19.423766 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:19.423722    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:20.935704 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:20.935658    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:22.705642 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:04:22.705601    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:04:26.073215 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:26.073174    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:32.932712 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:32.932618    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:04:38.836201 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:38.836165    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:04:39.601726 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:39.601683    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:40.982559 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:40.982498    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:46.997930 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:46.997866    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:04:47.931783 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:47.931747    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"]
Nov 02 16:04:47.931990 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:47.931970    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn" podUID="5205492e-f380-48ef-9879-f03eaead7bcb" containerName="manager" containerID="cri-o://a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22" gracePeriod=10
Nov 02 16:04:48.201530 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:48.201499    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 16:04:48.315760 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:48.315724    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5205492e-f380-48ef-9879-f03eaead7bcb-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5205492e-f380-48ef-9879-f03eaead7bcb-extensions-socket-volume\") pod \"5205492e-f380-48ef-9879-f03eaead7bcb\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") "
Nov 02 16:04:48.315760 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:48.315765    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5205492e-f380-48ef-9879-f03eaead7bcb-kube-api-access-hfrpd\" (UniqueName: \"kubernetes.io/projected/5205492e-f380-48ef-9879-f03eaead7bcb-kube-api-access-hfrpd\") pod \"5205492e-f380-48ef-9879-f03eaead7bcb\" (UID: \"5205492e-f380-48ef-9879-f03eaead7bcb\") "
Nov 02 16:04:48.316138 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:48.316111    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5205492e-f380-48ef-9879-f03eaead7bcb-extensions-socket-volume" pod "5205492e-f380-48ef-9879-f03eaead7bcb" (UID: "5205492e-f380-48ef-9879-f03eaead7bcb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:04:48.318178 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:48.318137    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5205492e-f380-48ef-9879-f03eaead7bcb-kube-api-access-hfrpd" pod "5205492e-f380-48ef-9879-f03eaead7bcb" (UID: "5205492e-f380-48ef-9879-f03eaead7bcb"). InnerVolumeSpecName "kube-api-access-hfrpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:04:48.417026 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:48.416978    7019 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5205492e-f380-48ef-9879-f03eaead7bcb-extensions-socket-volume\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:04:48.417026 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:48.417019    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfrpd\" (UniqueName: \"kubernetes.io/projected/5205492e-f380-48ef-9879-f03eaead7bcb-kube-api-access-hfrpd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:04:49.022898 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.022858    7019 generic.go:356] "Generic (PLEG): container finished" podID="5205492e-f380-48ef-9879-f03eaead7bcb" containerID="a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22" exitCode=0
Nov 02 16:04:49.023103 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.022918    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"
Nov 02 16:04:49.023103 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.022949    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn" event={"ID":"5205492e-f380-48ef-9879-f03eaead7bcb","Type":"ContainerDied","Data":"a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22"}
Nov 02 16:04:49.023103 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.023004    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn" event={"ID":"5205492e-f380-48ef-9879-f03eaead7bcb","Type":"ContainerDied","Data":"aca25aeb235d47c69e61ec1bf7b6f360271c9c6eef445431b3ad59873b29fdc3"}
Nov 02 16:04:49.023103 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.023027    7019 scope.go:122] "RemoveContainer" containerID="a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22"
Nov 02 16:04:49.037318 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.037293    7019 scope.go:122] "RemoveContainer" containerID="a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22"
Nov 02 16:04:49.037728 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:04:49.037709    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22\": container with ID starting with a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22 not found: ID does not exist" containerID="a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22"
Nov 02 16:04:49.037803 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.037737    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22"} err="failed to get container status \"a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22\": rpc error: code = NotFound desc = could not find container \"a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22\": container with ID starting with a978b2f96096585abca165b7eb571af2967600f066b2d2d697c7f2418a745e22 not found: ID does not exist"
Nov 02 16:04:49.090632 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.090594    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"]
Nov 02 16:04:49.103708 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.103669    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6q2hn"]
Nov 02 16:04:49.860592 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:04:49.860557    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5205492e-f380-48ef-9879-f03eaead7bcb" path="/var/lib/kubelet/pods/5205492e-f380-48ef-9879-f03eaead7bcb/volumes"
Nov 02 16:04:52.703629 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:04:52.703591    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:05:00.373213 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:00.373176    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:02.407749 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:02.407708    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:12.058099 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:12.058058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:21.061508 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:21.061471    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:22.704713 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:05:22.704673    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:05:26.194305 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:26.194268    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:30.991181 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:30.991142    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:36.782540 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:36.782498    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:36.935377 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:36.935335    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:41.643917 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:41.643877    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:05:51.084422 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.084382    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"]
Nov 02 16:05:51.084890 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.084851    7019 cpu_manager.go:414] "containerMap: removing container" podUID="ed41b7fd-f044-40c0-a392-8da7588dc5f1" containerName="revoke-keys"
Nov 02 16:05:51.084890 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.084867    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed41b7fd-f044-40c0-a392-8da7588dc5f1" containerName="revoke-keys"
Nov 02 16:05:51.084990 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.084893    7019 cpu_manager.go:414] "containerMap: removing container" podUID="5205492e-f380-48ef-9879-f03eaead7bcb" containerName="manager"
Nov 02 16:05:51.084990 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.084899    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5205492e-f380-48ef-9879-f03eaead7bcb" containerName="manager"
Nov 02 16:05:51.084990 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.084969    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed41b7fd-f044-40c0-a392-8da7588dc5f1" containerName="revoke-keys"
Nov 02 16:05:51.084990 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.084982    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="5205492e-f380-48ef-9879-f03eaead7bcb" containerName="manager"
Nov 02 16:05:51.088415 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.088398    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.103162 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.103138    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"]
Nov 02 16:05:51.114418 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.114393    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rdv4c\""
Nov 02 16:05:51.220365 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.220319    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9n2b\" (UniqueName: \"kubernetes.io/projected/952f1fec-5220-4de1-8107-9de8bb4d7fea-kube-api-access-j9n2b\") pod \"kuadrant-operator-controller-manager-678d87766f-g4sbf\" (UID: \"952f1fec-5220-4de1-8107-9de8bb4d7fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.220365 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.220368    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952f1fec-5220-4de1-8107-9de8bb4d7fea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g4sbf\" (UID: \"952f1fec-5220-4de1-8107-9de8bb4d7fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.321049 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.321007    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9n2b\" (UniqueName: \"kubernetes.io/projected/952f1fec-5220-4de1-8107-9de8bb4d7fea-kube-api-access-j9n2b\") pod \"kuadrant-operator-controller-manager-678d87766f-g4sbf\" (UID: \"952f1fec-5220-4de1-8107-9de8bb4d7fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.321049 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.321052    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952f1fec-5220-4de1-8107-9de8bb4d7fea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g4sbf\" (UID: \"952f1fec-5220-4de1-8107-9de8bb4d7fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.321507 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.321488    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/952f1fec-5220-4de1-8107-9de8bb4d7fea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g4sbf\" (UID: \"952f1fec-5220-4de1-8107-9de8bb4d7fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.345803 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.345732    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9n2b\" (UniqueName: \"kubernetes.io/projected/952f1fec-5220-4de1-8107-9de8bb4d7fea-kube-api-access-j9n2b\") pod \"kuadrant-operator-controller-manager-678d87766f-g4sbf\" (UID: \"952f1fec-5220-4de1-8107-9de8bb4d7fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.401858 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.401826    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:51.728310 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:51.728280    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"]
Nov 02 16:05:51.728437 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:05:51.728348    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952f1fec_5220_4de1_8107_9de8bb4d7fea.slice/crio-3f181bf19f4f77665d5f3f3fd895a0ca46af85b439dacc3e7d33881925d19c9f WatchSource:0}: Error finding container 3f181bf19f4f77665d5f3f3fd895a0ca46af85b439dacc3e7d33881925d19c9f: Status 404 returned error can't find the container with id 3f181bf19f4f77665d5f3f3fd895a0ca46af85b439dacc3e7d33881925d19c9f
Nov 02 16:05:52.345051 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:52.345012    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf" event={"ID":"952f1fec-5220-4de1-8107-9de8bb4d7fea","Type":"ContainerStarted","Data":"8ed7fbd024b54ec76bf6b3e91cb988487eb562253860eb8c583e08d2553cfd94"}
Nov 02 16:05:52.345472 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:52.345088    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf" event={"ID":"952f1fec-5220-4de1-8107-9de8bb4d7fea","Type":"ContainerStarted","Data":"3f181bf19f4f77665d5f3f3fd895a0ca46af85b439dacc3e7d33881925d19c9f"}
Nov 02 16:05:52.345472 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:52.345129    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:05:52.416922 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:52.416873    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf" podStartSLOduration=1.416861075 podStartE2EDuration="1.416861075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:05:52.403666035 +0000 UTC m=+4146.657619650" watchObservedRunningTime="2026-11-02 16:05:52.416861075 +0000 UTC m=+4146.670814696"
Nov 02 16:05:52.704465 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:05:52.704422    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:05:53.786950 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:05:53.786894    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:02.527459 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:02.527379    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:03.352249 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:03.352212    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g4sbf"
Nov 02 16:06:09.252249 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:09.252213    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:06:21.157488 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:21.157449    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:22.704039 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:06:22.703995    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:06:24.558248 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:24.558207    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:06:26.332342 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:26.332305    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:30.008672 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:30.006310    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:06:40.564302 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:40.564256    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:40.716110 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:40.716078    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:44.149943 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:44.149908    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:46.042552 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:46.042510    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:06:46.519251 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:46.519211    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:47.069057 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:47.069020    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:47.227198 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:47.227162    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:51.034446 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:51.034405    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:51.187187 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:51.187145    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:52.573267 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:52.573229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:52.708418 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:06:52.708378    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:06:52.716893 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:52.716861    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:52.865656 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:52.865623    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:53.021217 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:53.021177    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:53.291167 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:53.291070    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:54.412154 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:54.412122    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:56.123686 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:56.123628    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:56.363361 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:56.363327    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:06:58.046817 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:58.046781    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:06:59.939011 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:06:59.938967    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:01.731777 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:01.731736    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:02.662184 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:02.662135    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:03.608453 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:03.608419    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:05.306168 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:05.306132    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:05.895369 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:05.895337    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:06.543080 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:06.543044    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:06.712533 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:06.712499    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:07.344766 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:07.344727    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:09.202039 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:09.201995    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:09.353691 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:09.353633    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:11.220676 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:11.220631    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:11.653409 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:11.653372    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:07:13.103899 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:13.103856    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:14.894363 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:14.894322    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:16.618228 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:16.618189    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:18.417290 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:18.417254    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:20.118891 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:20.118855    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:21.325368 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:21.325334    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:22.008145 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:22.008118    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:22.704279 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:07:22.704240    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:07:23.721158 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:23.721124    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:25.504851 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:25.504799    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:26.449486 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:26.449451    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:26.674877 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:26.674846    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:07:27.363944 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:27.363908    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:29.127145 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:29.127107    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:30.820466 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:30.820426    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:32.592187 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:32.592150    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:34.401990 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:34.401949    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:36.197566 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:36.197528    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:38.014391 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:38.014354    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:39.719186 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:39.719142    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:40.581318 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.581283    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"]
Nov 02 16:07:40.585212 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.585191    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.602585 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.602560    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:07:40.614951 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.614926    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:07:40.616545 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.616526    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"]
Nov 02 16:07:40.663903 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.663870    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.663903 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.663906    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.664114 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.663926    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5crd\" (UniqueName: \"kubernetes.io/projected/878261e4-6245-4a82-b67a-fe3fa159c4de-kube-api-access-b5crd\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.764478 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.764441    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.764478 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.764481    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.765052 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.764506    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5crd\" (UniqueName: \"kubernetes.io/projected/878261e4-6245-4a82-b67a-fe3fa159c4de-kube-api-access-b5crd\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.765345 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.765318    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.767016 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.766997    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.789201 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.789169    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5crd\" (UniqueName: \"kubernetes.io/projected/878261e4-6245-4a82-b67a-fe3fa159c4de-kube-api-access-b5crd\") pod \"maas-api-revoke-sub-a3dad175-mnshr\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:40.898326 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:40.898295    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:41.198534 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:41.198502    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"]
Nov 02 16:07:41.198807 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:07:41.198781    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878261e4_6245_4a82_b67a_fe3fa159c4de.slice/crio-dc1275f183bbad82104a55c903cf10505124818e37f4ab2917c5b9cf9c69b972 WatchSource:0}: Error finding container dc1275f183bbad82104a55c903cf10505124818e37f4ab2917c5b9cf9c69b972: Status 404 returned error can't find the container with id dc1275f183bbad82104a55c903cf10505124818e37f4ab2917c5b9cf9c69b972
Nov 02 16:07:41.518788 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:41.518756    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:41.675974 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:41.675935    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:41.920622 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:41.920584    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr" event={"ID":"878261e4-6245-4a82-b67a-fe3fa159c4de","Type":"ContainerStarted","Data":"1193798591f4ed38e6f95767f87540e56cac901c956841b33e0da60f52f17c92"}
Nov 02 16:07:41.921088 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:41.920635    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr" event={"ID":"878261e4-6245-4a82-b67a-fe3fa159c4de","Type":"ContainerStarted","Data":"dc1275f183bbad82104a55c903cf10505124818e37f4ab2917c5b9cf9c69b972"}
Nov 02 16:07:41.964063 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:41.964010    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr" podStartSLOduration=1.963997827 podStartE2EDuration="1.963997827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:07:41.962207605 +0000 UTC m=+4256.216161229" watchObservedRunningTime="2026-11-02 16:07:41.963997827 +0000 UTC m=+4256.217951450"
Nov 02 16:07:42.684346 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:42.684307    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:42.926928 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:42.926898    7019 generic.go:356] "Generic (PLEG): container finished" podID="878261e4-6245-4a82-b67a-fe3fa159c4de" containerID="1193798591f4ed38e6f95767f87540e56cac901c956841b33e0da60f52f17c92" exitCode=0
Nov 02 16:07:42.927370 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:42.926962    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr" event={"ID":"878261e4-6245-4a82-b67a-fe3fa159c4de","Type":"ContainerDied","Data":"1193798591f4ed38e6f95767f87540e56cac901c956841b33e0da60f52f17c92"}
Nov 02 16:07:43.335299 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:43.335254    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:44.068014 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.067987    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:44.097666 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.097617    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/878261e4-6245-4a82-b67a-fe3fa159c4de-kube-api-access-b5crd\" (UniqueName: \"kubernetes.io/projected/878261e4-6245-4a82-b67a-fe3fa159c4de-kube-api-access-b5crd\") pod \"878261e4-6245-4a82-b67a-fe3fa159c4de\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") "
Nov 02 16:07:44.097805 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.097793    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-openshift-ca\") pod \"878261e4-6245-4a82-b67a-fe3fa159c4de\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") "
Nov 02 16:07:44.097856 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.097826    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-xks-ca\") pod \"878261e4-6245-4a82-b67a-fe3fa159c4de\" (UID: \"878261e4-6245-4a82-b67a-fe3fa159c4de\") "
Nov 02 16:07:44.098176 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.098148    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-openshift-ca" pod "878261e4-6245-4a82-b67a-fe3fa159c4de" (UID: "878261e4-6245-4a82-b67a-fe3fa159c4de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:07:44.099744 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.099718    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/878261e4-6245-4a82-b67a-fe3fa159c4de-kube-api-access-b5crd" pod "878261e4-6245-4a82-b67a-fe3fa159c4de" (UID: "878261e4-6245-4a82-b67a-fe3fa159c4de"). InnerVolumeSpecName "kube-api-access-b5crd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:07:44.099866 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.099796    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-xks-ca" pod "878261e4-6245-4a82-b67a-fe3fa159c4de" (UID: "878261e4-6245-4a82-b67a-fe3fa159c4de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:07:44.199309 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.199223    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:07:44.199309 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.199252    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/878261e4-6245-4a82-b67a-fe3fa159c4de-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:07:44.199309 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.199262    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5crd\" (UniqueName: \"kubernetes.io/projected/878261e4-6245-4a82-b67a-fe3fa159c4de-kube-api-access-b5crd\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:07:44.938157 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.938119    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr" event={"ID":"878261e4-6245-4a82-b67a-fe3fa159c4de","Type":"ContainerDied","Data":"dc1275f183bbad82104a55c903cf10505124818e37f4ab2917c5b9cf9c69b972"}
Nov 02 16:07:44.938157 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.938144    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"
Nov 02 16:07:44.938380 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:44.938181    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc1275f183bbad82104a55c903cf10505124818e37f4ab2917c5b9cf9c69b972"
Nov 02 16:07:45.012138 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:45.012099    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:46.797317 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:46.797271    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:48.599862 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:48.599828    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:50.314445 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:50.314412    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:51.467857 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.467819    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:51.668960 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.668927    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"]
Nov 02 16:07:51.669368 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.669354    7019 cpu_manager.go:414] "containerMap: removing container" podUID="878261e4-6245-4a82-b67a-fe3fa159c4de" containerName="revoke-keys"
Nov 02 16:07:51.669416 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.669369    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="878261e4-6245-4a82-b67a-fe3fa159c4de" containerName="revoke-keys"
Nov 02 16:07:51.669453 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.669448    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="878261e4-6245-4a82-b67a-fe3fa159c4de" containerName="revoke-keys"
Nov 02 16:07:51.672322 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.672307    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.692287 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.692258    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:07:51.701999 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.701967    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m5926\""
Nov 02 16:07:51.704810 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.704780    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"]
Nov 02 16:07:51.769741 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.769622    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbldr\" (UniqueName: \"kubernetes.io/projected/9229b576-b8a5-441a-83e6-f20cc63ee98b-kube-api-access-jbldr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.769905 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.769745    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.769905 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.769792    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.872858 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.872821    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.873229 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.873211    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbldr\" (UniqueName: \"kubernetes.io/projected/9229b576-b8a5-441a-83e6-f20cc63ee98b-kube-api-access-jbldr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.873344 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.873331    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.874305 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.874279    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.879156 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.879127    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.899018 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.898985    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbldr\" (UniqueName: \"kubernetes.io/projected/9229b576-b8a5-441a-83e6-f20cc63ee98b-kube-api-access-jbldr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:51.985509 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:51.985477    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:52.238076 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:52.238039    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:07:52.290897 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:52.289705    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"]
Nov 02 16:07:52.295107 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:07:52.295061    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9229b576_b8a5_441a_83e6_f20cc63ee98b.slice/crio-71592bfab37403ae2c67c176574f8cf8d2347e02ea2f711865cdb0b1b9d94acb WatchSource:0}: Error finding container 71592bfab37403ae2c67c176574f8cf8d2347e02ea2f711865cdb0b1b9d94acb: Status 404 returned error can't find the container with id 71592bfab37403ae2c67c176574f8cf8d2347e02ea2f711865cdb0b1b9d94acb
Nov 02 16:07:52.305667 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:52.305630    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:52.704394 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:07:52.704358    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:07:52.981920 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:52.981883    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp" event={"ID":"9229b576-b8a5-441a-83e6-f20cc63ee98b","Type":"ContainerStarted","Data":"8900954e15fcaf188371062a2cf175d131bb1e21556356bb3b1210f3f834bb91"}
Nov 02 16:07:52.982095 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:52.981940    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp" event={"ID":"9229b576-b8a5-441a-83e6-f20cc63ee98b","Type":"ContainerStarted","Data":"71592bfab37403ae2c67c176574f8cf8d2347e02ea2f711865cdb0b1b9d94acb"}
Nov 02 16:07:53.024376 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:53.024324    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp" podStartSLOduration=2.02431111 podStartE2EDuration="2.02431111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:07:53.02329458 +0000 UTC m=+4267.277248202" watchObservedRunningTime="2026-11-02 16:07:53.02431111 +0000 UTC m=+4267.278264733"
Nov 02 16:07:53.988388 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:53.988352    7019 generic.go:356] "Generic (PLEG): container finished" podID="9229b576-b8a5-441a-83e6-f20cc63ee98b" containerID="8900954e15fcaf188371062a2cf175d131bb1e21556356bb3b1210f3f834bb91" exitCode=0
Nov 02 16:07:53.988839 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:53.988395    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp" event={"ID":"9229b576-b8a5-441a-83e6-f20cc63ee98b","Type":"ContainerDied","Data":"8900954e15fcaf188371062a2cf175d131bb1e21556356bb3b1210f3f834bb91"}
Nov 02 16:07:54.212555 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:54.212506    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:55.132587 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.132564    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:55.206499 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.206468    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-openshift-ca\") pod \"9229b576-b8a5-441a-83e6-f20cc63ee98b\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") "
Nov 02 16:07:55.206698 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.206505    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9229b576-b8a5-441a-83e6-f20cc63ee98b-kube-api-access-jbldr\" (UniqueName: \"kubernetes.io/projected/9229b576-b8a5-441a-83e6-f20cc63ee98b-kube-api-access-jbldr\") pod \"9229b576-b8a5-441a-83e6-f20cc63ee98b\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") "
Nov 02 16:07:55.206698 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.206553    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-xks-ca\") pod \"9229b576-b8a5-441a-83e6-f20cc63ee98b\" (UID: \"9229b576-b8a5-441a-83e6-f20cc63ee98b\") "
Nov 02 16:07:55.206862 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.206840    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-openshift-ca" pod "9229b576-b8a5-441a-83e6-f20cc63ee98b" (UID: "9229b576-b8a5-441a-83e6-f20cc63ee98b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:07:55.208680 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.208631    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-xks-ca" pod "9229b576-b8a5-441a-83e6-f20cc63ee98b" (UID: "9229b576-b8a5-441a-83e6-f20cc63ee98b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:07:55.208798 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.208782    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9229b576-b8a5-441a-83e6-f20cc63ee98b-kube-api-access-jbldr" pod "9229b576-b8a5-441a-83e6-f20cc63ee98b" (UID: "9229b576-b8a5-441a-83e6-f20cc63ee98b"). InnerVolumeSpecName "kube-api-access-jbldr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:07:55.308108 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.308028    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:07:55.308108 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.308056    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbldr\" (UniqueName: \"kubernetes.io/projected/9229b576-b8a5-441a-83e6-f20cc63ee98b-kube-api-access-jbldr\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:07:55.308108 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:55.308065    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9229b576-b8a5-441a-83e6-f20cc63ee98b-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:07:56.000824 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:56.000790    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"
Nov 02 16:07:56.000824 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:56.000810    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp" event={"ID":"9229b576-b8a5-441a-83e6-f20cc63ee98b","Type":"ContainerDied","Data":"71592bfab37403ae2c67c176574f8cf8d2347e02ea2f711865cdb0b1b9d94acb"}
Nov 02 16:07:56.001075 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:56.000869    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71592bfab37403ae2c67c176574f8cf8d2347e02ea2f711865cdb0b1b9d94acb"
Nov 02 16:07:56.012023 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:56.011992    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:56.552740 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:56.552707    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:57.710079 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:57.710040    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:07:59.546636 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:07:59.546599    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:01.295178 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:01.295140    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:02.307478 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:02.307440    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:02.467152 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:02.467110    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:02.760719 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:02.760636    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:03.002883 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:03.002845    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:04.541273 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:04.541233    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:05.075729 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:05.075691    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:05.181559 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:05.181522    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"]
Nov 02 16:08:05.194938 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:05.194902    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-bbe3a8-54xvp"]
Nov 02 16:08:05.219748 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:05.219717    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:05.704187 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:05.704153    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:05.861536 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:05.861500    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9229b576-b8a5-441a-83e6-f20cc63ee98b" path="/var/lib/kubelet/pods/9229b576-b8a5-441a-83e6-f20cc63ee98b/volumes"
Nov 02 16:08:07.039370 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:07.039299    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:08:11.741586 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:11.741517    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:11.884199 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:11.884165    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:12.110185 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:12.110075    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:21.431830 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:21.431787    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:22.704635 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:08:22.704598    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:08:26.068449 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:26.068404    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:26.471269 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:26.471228    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:26.630549 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:26.630515    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:28.718318 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:28.718272    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:08:33.127980 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:33.127943    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:34.680994 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:34.680959    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:41.002537 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:41.002498    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:41.043556 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:41.043524    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"]
Nov 02 16:08:41.056242 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:41.056211    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25ac2ab-qz2j2"]
Nov 02 16:08:41.860410 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:41.860373    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a284242a-b7f7-4ef3-8626-28301d724b4e" path="/var/lib/kubelet/pods/a284242a-b7f7-4ef3-8626-28301d724b4e/volumes"
Nov 02 16:08:43.221420 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:43.221385    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:08:45.656695 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:08:45.656643    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:08:52.705207 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:08:52.705168    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:09:01.362302 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:01.362214    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:09:11.304541 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:11.304503    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:09:13.447363 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:13.447330    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:09:13.452820 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:13.452785    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:09:20.373841 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:20.373188    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:09:21.556716 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:21.556680    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:09:22.704872 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:09:22.704832    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:09:26.703090 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:26.703051    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:09:28.652441 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:28.652403    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:09:35.742880 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:35.742842    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:09:40.076664 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:40.076618    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:09:40.220656 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:40.220615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:09:52.016936 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:52.016899    7019 scope.go:122] "RemoveContainer" containerID="e599cfbe208f7537ac2fa01d4e812add724b0b1848c9e95312aef4c9d1120d5c"
Nov 02 16:09:52.048075 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:52.048047    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:09:52.704583 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:09:52.704544    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:09:59.842076 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:09:59.842031    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:00.208290 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.208240    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"]
Nov 02 16:10:00.209253 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.209220    7019 cpu_manager.go:414] "containerMap: removing container" podUID="9229b576-b8a5-441a-83e6-f20cc63ee98b" containerName="revoke-keys"
Nov 02 16:10:00.209253 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.209252    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9229b576-b8a5-441a-83e6-f20cc63ee98b" containerName="revoke-keys"
Nov 02 16:10:00.209425 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.209412    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="9229b576-b8a5-441a-83e6-f20cc63ee98b" containerName="revoke-keys"
Nov 02 16:10:00.213062 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.213041    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.234549 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.234524    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2fa6473d-openshift-default-dockercfg-zw92v\""
Nov 02 16:10:00.247037 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.247002    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"]
Nov 02 16:10:00.290640 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.290601    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-envoy\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.290835 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.290726    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgz7t\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-kube-api-access-mgz7t\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.290835 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.290768    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44ae50f-9b0f-40b4-af59-24170809ca56-istiod-ca-cert\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.290835 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.290794    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-token\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.290835 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.290823    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-podinfo\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.291041 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.290953    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-certs\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.291041 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.290992    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-data\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.291140 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.291073    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-credential-socket\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.291189 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.291144    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-socket\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.309297 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.309263    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:00.393554 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.393471    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgz7t\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-kube-api-access-mgz7t\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.394297 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.394267    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44ae50f-9b0f-40b4-af59-24170809ca56-istiod-ca-cert\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.395612 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.395314    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44ae50f-9b0f-40b4-af59-24170809ca56-istiod-ca-cert\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.395825 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.395794    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-token\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.396855 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.396834    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-podinfo\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.397596 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.397576    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-certs\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.398072 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.398054    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-data\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.398553 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.398536    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-credential-socket\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.399018 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.398970    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-socket\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.399227 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.399211    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-envoy\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.399419 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.398918    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-credential-socket\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.399953 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.399780    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-socket\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.400128 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.398468    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-data\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.400261 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.398016    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-certs\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.404153 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.404048    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-envoy\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.408001 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.407960    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-podinfo\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.443401 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.443368    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgz7t\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-kube-api-access-mgz7t\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.446138 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.446116    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-token\") pod \"e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") " pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.520057 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.519976    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:00.529821 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.529792    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:00.846896 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:10:00.846866    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44ae50f_9b0f_40b4_af59_24170809ca56.slice/crio-887b3003b10e6e2b3b724d8c93c59958b589f986474ae0d624dc4d642f57f90f WatchSource:0}: Error finding container 887b3003b10e6e2b3b724d8c93c59958b589f986474ae0d624dc4d642f57f90f: Status 404 returned error can't find the container with id 887b3003b10e6e2b3b724d8c93c59958b589f986474ae0d624dc4d642f57f90f
Nov 02 16:10:00.847360 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.847318    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"]
Nov 02 16:10:00.849390 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.849359    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:00.849471 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.849426    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:00.849471 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:00.849454    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:01.804452 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:01.804414    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" event={"ID":"c44ae50f-9b0f-40b4-af59-24170809ca56","Type":"ContainerStarted","Data":"dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff"}
Nov 02 16:10:01.804661 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:01.804480    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" event={"ID":"c44ae50f-9b0f-40b4-af59-24170809ca56","Type":"ContainerStarted","Data":"887b3003b10e6e2b3b724d8c93c59958b589f986474ae0d624dc4d642f57f90f"}
Nov 02 16:10:01.864455 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:01.864404    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podStartSLOduration=1.864391914 podStartE2EDuration="1.864391914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:01.863943343 +0000 UTC m=+4396.117897142" watchObservedRunningTime="2026-11-02 16:10:01.864391914 +0000 UTC m=+4396.118345538"
Nov 02 16:10:02.530184 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:02.530153    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:02.534930 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:02.534901    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:02.809845 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:02.809759    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:02.810849 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:02.810825    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:03.625023 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:03.624988    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:03.792397 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:03.792361    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:05.313271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:05.313101    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:05.493884 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:05.493850    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:05.683907 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:05.683869    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:07.304986 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:07.304956    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:07.898460 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:07.898420    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:09.011833 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:09.011720    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:09.226764 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:09.226726    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:10:10.807882 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:10.807808    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:12.500317 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:12.500280    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:14.212697 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:14.212662    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:16.113572 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:16.113523    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:17.894624 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:17.894590    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:18.801427 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.800879    7019 patch_prober.go:29] interesting pod/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": read tcp 10.134.0.2:44272->10.134.1.59:15021: read: connection reset by peer" start-of-body=
Nov 02 16:10:18.801427 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.800952    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": read tcp 10.134.0.2:44272->10.134.1.59:15021: read: connection reset by peer"
Nov 02 16:10:18.920787 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.920751    7019 generic.go:356] "Generic (PLEG): container finished" podID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerID="dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff" exitCode=0
Nov 02 16:10:18.921203 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.920823    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" event={"ID":"c44ae50f-9b0f-40b4-af59-24170809ca56","Type":"ContainerDied","Data":"dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff"}
Nov 02 16:10:18.921203 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.921176    7019 scope.go:122] "RemoveContainer" containerID="dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff"
Nov 02 16:10:18.922428 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.922400    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:18.922510 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.922468    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:18.922510 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:18.922500    7019 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 02 16:10:19.722345 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:19.722306    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:19.928033 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:19.927997    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" event={"ID":"c44ae50f-9b0f-40b4-af59-24170809ca56","Type":"ContainerStarted","Data":"730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312"}
Nov 02 16:10:20.305394 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.305349    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"]
Nov 02 16:10:20.308321 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.308296    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.324091 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.324063    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:10:20.324208 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.324133    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8rvm8\""
Nov 02 16:10:20.328883 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.328862    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"]
Nov 02 16:10:20.413991 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.413958    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.414178 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.414108    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8kn\" (UniqueName: \"kubernetes.io/projected/b829eb9d-82df-496f-90ce-e270e714febf-kube-api-access-4s8kn\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.414178 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.414139    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.515500 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.515469    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8kn\" (UniqueName: \"kubernetes.io/projected/b829eb9d-82df-496f-90ce-e270e714febf-kube-api-access-4s8kn\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.515752 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.515507    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.515752 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.515552    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.516204 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.516174    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.517925 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.517902    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.530804 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.530780    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:20.530975 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.530952    7019 patch_prober.go:29] interesting pod/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 16:10:20.531025 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.531004    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 16:10:20.540090 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.540062    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8kn\" (UniqueName: \"kubernetes.io/projected/b829eb9d-82df-496f-90ce-e270e714febf-kube-api-access-4s8kn\") pod \"maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.620283 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.620248    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:20.941948 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:20.941922    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"]
Nov 02 16:10:20.942999 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:10:20.942969    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb829eb9d_82df_496f_90ce_e270e714febf.slice/crio-97c03d23abd1a7447f950b24e31dfe34def8b749eff6bcc0a460ba2fdad4e470 WatchSource:0}: Error finding container 97c03d23abd1a7447f950b24e31dfe34def8b749eff6bcc0a460ba2fdad4e470: Status 404 returned error can't find the container with id 97c03d23abd1a7447f950b24e31dfe34def8b749eff6bcc0a460ba2fdad4e470
Nov 02 16:10:21.503314 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:21.503208    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:21.530566 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:21.530513    7019 patch_prober.go:29] interesting pod/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 16:10:21.530800 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:21.530580    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 16:10:21.690945 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:21.690723    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:21.947686 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:21.947625    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n" event={"ID":"b829eb9d-82df-496f-90ce-e270e714febf","Type":"ContainerStarted","Data":"3dd400e590d882ad502727eda56637920bc83813f0b788a61c84700722276070"}
Nov 02 16:10:21.948141 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:21.947732    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n" event={"ID":"b829eb9d-82df-496f-90ce-e270e714febf","Type":"ContainerStarted","Data":"97c03d23abd1a7447f950b24e31dfe34def8b749eff6bcc0a460ba2fdad4e470"}
Nov 02 16:10:21.991887 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:21.991306    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n" podStartSLOduration=1.991286288 podStartE2EDuration="1.991286288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:10:21.990228394 +0000 UTC m=+4416.244182020" watchObservedRunningTime="2026-11-02 16:10:21.991286288 +0000 UTC m=+4416.245239914"
Nov 02 16:10:22.530809 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:22.530766    7019 patch_prober.go:29] interesting pod/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 16:10:22.531028 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:22.530828    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 16:10:22.713755 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:10:22.713708    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:10:22.953979 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:22.953939    7019 generic.go:356] "Generic (PLEG): container finished" podID="b829eb9d-82df-496f-90ce-e270e714febf" containerID="3dd400e590d882ad502727eda56637920bc83813f0b788a61c84700722276070" exitCode=0
Nov 02 16:10:22.954397 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:22.954020    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n" event={"ID":"b829eb9d-82df-496f-90ce-e270e714febf","Type":"ContainerDied","Data":"3dd400e590d882ad502727eda56637920bc83813f0b788a61c84700722276070"}
Nov 02 16:10:23.312563 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:23.312467    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:23.531134 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:23.531092    7019 patch_prober.go:29] interesting pod/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 16:10:23.531338 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:23.531158    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 16:10:24.156149 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.156114    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:24.258271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.258184    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-openshift-ca\") pod \"b829eb9d-82df-496f-90ce-e270e714febf\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") "
Nov 02 16:10:24.258271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.258227    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-xks-ca\") pod \"b829eb9d-82df-496f-90ce-e270e714febf\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") "
Nov 02 16:10:24.258271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.258262    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b829eb9d-82df-496f-90ce-e270e714febf-kube-api-access-4s8kn\" (UniqueName: \"kubernetes.io/projected/b829eb9d-82df-496f-90ce-e270e714febf-kube-api-access-4s8kn\") pod \"b829eb9d-82df-496f-90ce-e270e714febf\" (UID: \"b829eb9d-82df-496f-90ce-e270e714febf\") "
Nov 02 16:10:24.258582 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.258554    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-openshift-ca" pod "b829eb9d-82df-496f-90ce-e270e714febf" (UID: "b829eb9d-82df-496f-90ce-e270e714febf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:10:24.258716 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.258690    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:24.260486 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.260459    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-xks-ca" pod "b829eb9d-82df-496f-90ce-e270e714febf" (UID: "b829eb9d-82df-496f-90ce-e270e714febf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:10:24.262188 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.262162    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b829eb9d-82df-496f-90ce-e270e714febf-kube-api-access-4s8kn" pod "b829eb9d-82df-496f-90ce-e270e714febf" (UID: "b829eb9d-82df-496f-90ce-e270e714febf"). InnerVolumeSpecName "kube-api-access-4s8kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:24.360139 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.360098    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s8kn\" (UniqueName: \"kubernetes.io/projected/b829eb9d-82df-496f-90ce-e270e714febf-kube-api-access-4s8kn\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:24.360139 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.360129    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b829eb9d-82df-496f-90ce-e270e714febf-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:24.530445 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.530360    7019 patch_prober.go:29] interesting pod/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 16:10:24.530445 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.530412    7019 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 16:10:24.967633 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.967598    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n" event={"ID":"b829eb9d-82df-496f-90ce-e270e714febf","Type":"ContainerDied","Data":"97c03d23abd1a7447f950b24e31dfe34def8b749eff6bcc0a460ba2fdad4e470"}
Nov 02 16:10:24.967633 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.967638    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"
Nov 02 16:10:24.967890 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:24.967670    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97c03d23abd1a7447f950b24e31dfe34def8b749eff6bcc0a460ba2fdad4e470"
Nov 02 16:10:25.106291 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:25.106260    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:25.533795 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:25.533759    7019 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:25.534258 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:25.534031    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:25.534773 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:25.534755    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:26.803420 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:26.803383    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:26.951258 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:26.951182    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:28.515199 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:28.515159    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:29.945333 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:29.945298    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:30.105177 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:30.105137    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:31.187296 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:31.187261    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:31.299852 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:31.299814    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"]
Nov 02 16:10:31.311489 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:31.311456    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2fa6473d-r9s8n"]
Nov 02 16:10:31.333636 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:31.333599    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:31.861328 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:31.861297    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b829eb9d-82df-496f-90ce-e270e714febf" path="/var/lib/kubelet/pods/b829eb9d-82df-496f-90ce-e270e714febf/volumes"
Nov 02 16:10:33.279518 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:33.279484    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:10:35.529842 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:35.529807    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"]
Nov 02 16:10:35.530301 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:35.530076    7019 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" containerID="cri-o://730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312" gracePeriod=30
Nov 02 16:10:35.599943 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:35.599910    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:36.801147 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:36.801098    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:37.284072 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:37.284032    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:37.432366 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:37.432328    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:40.544059 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.544016    7019 patch_prober.go:29] interesting pod/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": EOF" start-of-body=
Nov 02 16:10:40.544500 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.544101    7019 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": EOF"
Nov 02 16:10:40.718764 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.718628    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:40.788532 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.788499    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:40.872861 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.872826    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:40.924715 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.924683    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-certs\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.924903 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.924760    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44ae50f-9b0f-40b4-af59-24170809ca56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44ae50f-9b0f-40b4-af59-24170809ca56-istiod-ca-cert\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.924970 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.924948    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-envoy\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.925032 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.924983    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-socket\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.925126 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925004    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-certs" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:10:40.925126 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925027    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-podinfo\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.925126 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925103    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-kube-api-access-mgz7t\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-kube-api-access-mgz7t\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.925308 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925154    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-token\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-token\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.925308 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925178    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44ae50f-9b0f-40b4-af59-24170809ca56-istiod-ca-cert" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:10:40.925308 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925223    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-credential-socket\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.925308 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925256    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-data\") pod \"c44ae50f-9b0f-40b4-af59-24170809ca56\" (UID: \"c44ae50f-9b0f-40b4-af59-24170809ca56\") "
Nov 02 16:10:40.925308 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925292    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-socket" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:10:40.925578 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925478    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-credential-socket" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:10:40.925636 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925601    7019 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:40.925636 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925623    7019 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-credential-socket\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:40.925782 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925636    7019 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-workload-certs\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:40.925782 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925636    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-data" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:10:40.925782 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.925682    7019 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c44ae50f-9b0f-40b4-af59-24170809ca56-istiod-ca-cert\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:40.927289 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.927264    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-podinfo" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 16:10:40.927414 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.927314    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-envoy" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 16:10:40.927414 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.927381    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-token" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:40.927414 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:40.927406    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-kube-api-access-mgz7t" pod "c44ae50f-9b0f-40b4-af59-24170809ca56" (UID: "c44ae50f-9b0f-40b4-af59-24170809ca56"). InnerVolumeSpecName "kube-api-access-mgz7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:10:41.026573 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.026540    7019 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-envoy\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:41.026573 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.026569    7019 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-podinfo\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:41.026573 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.026580    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgz7t\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-kube-api-access-mgz7t\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:41.026833 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.026589    7019 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-token\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:41.026833 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.026597    7019 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c44ae50f-9b0f-40b4-af59-24170809ca56-istio-data\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:10:41.054658 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.054623    7019 generic.go:356] "Generic (PLEG): container finished" podID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerID="730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312" exitCode=0
Nov 02 16:10:41.054836 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.054701    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"
Nov 02 16:10:41.054836 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.054710    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" event={"ID":"c44ae50f-9b0f-40b4-af59-24170809ca56","Type":"ContainerDied","Data":"730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312"}
Nov 02 16:10:41.054836 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.054770    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2" event={"ID":"c44ae50f-9b0f-40b4-af59-24170809ca56","Type":"ContainerDied","Data":"887b3003b10e6e2b3b724d8c93c59958b589f986474ae0d624dc4d642f57f90f"}
Nov 02 16:10:41.054836 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.054792    7019 scope.go:122] "RemoveContainer" containerID="730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312"
Nov 02 16:10:41.065015 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.064997    7019 scope.go:122] "RemoveContainer" containerID="dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff"
Nov 02 16:10:41.073340 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.073324    7019 scope.go:122] "RemoveContainer" containerID="730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312"
Nov 02 16:10:41.073581 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:10:41.073565    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312\": container with ID starting with 730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312 not found: ID does not exist" containerID="730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312"
Nov 02 16:10:41.073637 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.073591    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312"} err="failed to get container status \"730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312\": rpc error: code = NotFound desc = could not find container \"730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312\": container with ID starting with 730511e15379c1bd80b1348f08c3ce63778b2539584bf38980916162d5735312 not found: ID does not exist"
Nov 02 16:10:41.073637 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.073606    7019 scope.go:122] "RemoveContainer" containerID="dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff"
Nov 02 16:10:41.073829 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:10:41.073814    7019 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff\": container with ID starting with dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff not found: ID does not exist" containerID="dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff"
Nov 02 16:10:41.073875 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.073834    7019 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff"} err="failed to get container status \"dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff\": rpc error: code = NotFound desc = could not find container \"dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff\": container with ID starting with dc83203b3e0612c19fc1c39bbeeacde53821a2c1c00dd7d83a304872b0318aff not found: ID does not exist"
Nov 02 16:10:41.108780 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.108751    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"]
Nov 02 16:10:41.114252 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.114224    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2fa6473d-openshift-default-5c7f7b6d5f-9dmr2"]
Nov 02 16:10:41.860395 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:41.860364    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" path="/var/lib/kubelet/pods/c44ae50f-9b0f-40b4-af59-24170809ca56/volumes"
Nov 02 16:10:42.278347 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:42.278310    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:42.425784 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:42.425750    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:42.618887 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:42.618841    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:44.305374 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:44.305338    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:46.134643 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:46.134607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:46.144751 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:46.144719    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:10:47.946995 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:47.946959    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:49.917544 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:49.917505    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:51.738822 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:51.738778    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:52.040257 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:52.040161    7019 scope.go:122] "RemoveContainer" containerID="d0005a5c45d60c157a08aa0334958576e69aefcfad7ae9c37f8978012484ad77"
Nov 02 16:10:52.704696 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:10:52.704657    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:10:53.624422 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:53.624387    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:55.309245 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:55.309212    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:55.501387 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:55.501353    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:57.110733 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:57.110680    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:10:59.034675 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:10:59.034619    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:00.939273 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:00.939229    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:02.915165 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:02.915126    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:04.622792 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:04.622753    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:06.533829 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:06.533798    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:08.043905 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:08.043868    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:08.193546 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:08.193508    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:09.800895 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:09.800851    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:11.534469 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:11.534435    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:13.300170 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:13.300135    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:15.000134 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:15.000098    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:16.709780 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:16.709732    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:18.401760 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:18.401721    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:20.113672 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:20.113615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:21.820261 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:21.820221    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:21.968127 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:21.968092    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:22.705098 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:11:22.705055    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:11:23.695419 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:23.695380    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:25.412667 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:25.412603    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:26.946145 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:26.946111    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:27.117229 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:27.117189    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:28.910012 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:28.909979    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:30.698859 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:30.698820    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:32.508201 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:32.508157    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:34.203074 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:34.203037    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:35.926339 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:35.926301    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:37.706953 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:37.706920    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:39.502461 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:39.502421    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:41.200807 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:41.200777    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:43.011409 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:43.011365    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:44.809244 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:44.809215    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:46.506131 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:46.506092    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:48.298683 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:48.298634    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:49.998949 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:49.998916    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:50.685155 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:50.685115    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:50.886049 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:50.886012    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:51.803026 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:51.802991    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:52.704818 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:11:52.704781    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:11:53.502509 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:53.502473    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:55.188079 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:55.188045    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:56.935293 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:56.935193    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:11:58.622672 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:11:58.622607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:00.390670 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:00.390615    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:02.130032 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:02.129999    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:03.996931 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:03.996898    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:05.703246 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:05.703210    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:07.414860 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:07.414816    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:08.154807 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:08.154769    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:09.202801 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:09.202765    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:10.920861 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:10.920822    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:12.893277 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:12.893226    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:14.519290 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:14.519250    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:16.223725 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:16.223637    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:18.004457 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.004419    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:18.425140 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425102    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"]
Nov 02 16:12:18.425574 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425557    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy"
Nov 02 16:12:18.425641 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425575    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy"
Nov 02 16:12:18.425641 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425599    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b829eb9d-82df-496f-90ce-e270e714febf" containerName="revoke-keys"
Nov 02 16:12:18.425641 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425604    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b829eb9d-82df-496f-90ce-e270e714febf" containerName="revoke-keys"
Nov 02 16:12:18.425787 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425696    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy"
Nov 02 16:12:18.425787 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425708    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy"
Nov 02 16:12:18.425787 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.425719    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b829eb9d-82df-496f-90ce-e270e714febf" containerName="revoke-keys"
Nov 02 16:12:18.428904 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.428886    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.457192 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.457165    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8rvm8\""
Nov 02 16:12:18.458829 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.458808    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"]
Nov 02 16:12:18.480662 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.480621    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.480841 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.480820    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrn9\" (UniqueName: \"kubernetes.io/projected/2dd64c27-4a21-4e3a-a514-81a2c9da944e-kube-api-access-zdrn9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.480891 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.480862    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.544278 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.544245    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:12:18.581956 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.581921    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrn9\" (UniqueName: \"kubernetes.io/projected/2dd64c27-4a21-4e3a-a514-81a2c9da944e-kube-api-access-zdrn9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.582120 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.581970    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.582120 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.582005    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.582767 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.582736    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.584315 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.584293    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.605713 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.605684    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrn9\" (UniqueName: \"kubernetes.io/projected/2dd64c27-4a21-4e3a-a514-81a2c9da944e-kube-api-access-zdrn9\") pod \"maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:18.742219 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:18.742138    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:19.038666 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:19.038626    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"]
Nov 02 16:12:19.039285 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:12:19.039259    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd64c27_4a21_4e3a_a514_81a2c9da944e.slice/crio-7a39ded1156d32cf6b4e9b078a2006d6733ecde23a14558fa9dd94e3e19da464 WatchSource:0}: Error finding container 7a39ded1156d32cf6b4e9b078a2006d6733ecde23a14558fa9dd94e3e19da464: Status 404 returned error can't find the container with id 7a39ded1156d32cf6b4e9b078a2006d6733ecde23a14558fa9dd94e3e19da464
Nov 02 16:12:19.562537 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:19.562499    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt" event={"ID":"2dd64c27-4a21-4e3a-a514-81a2c9da944e","Type":"ContainerStarted","Data":"30e8d0e637e1d692d87a42a6b0de8f8db0edc84bbe0e38b55384b56996f14ee6"}
Nov 02 16:12:19.562781 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:19.562553    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt" event={"ID":"2dd64c27-4a21-4e3a-a514-81a2c9da944e","Type":"ContainerStarted","Data":"7a39ded1156d32cf6b4e9b078a2006d6733ecde23a14558fa9dd94e3e19da464"}
Nov 02 16:12:19.605187 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:19.605132    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt" podStartSLOduration=1.605117727 podStartE2EDuration="1.605117727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:12:19.604277729 +0000 UTC m=+4533.858231356" watchObservedRunningTime="2026-11-02 16:12:19.605117727 +0000 UTC m=+4533.859071350"
Nov 02 16:12:19.906635 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:19.906605    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:20.568912 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:20.568881    7019 generic.go:356] "Generic (PLEG): container finished" podID="2dd64c27-4a21-4e3a-a514-81a2c9da944e" containerID="30e8d0e637e1d692d87a42a6b0de8f8db0edc84bbe0e38b55384b56996f14ee6" exitCode=0
Nov 02 16:12:20.569371 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:20.568937    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt" event={"ID":"2dd64c27-4a21-4e3a-a514-81a2c9da944e","Type":"ContainerDied","Data":"30e8d0e637e1d692d87a42a6b0de8f8db0edc84bbe0e38b55384b56996f14ee6"}
Nov 02 16:12:21.707416 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.707388    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:21.736384 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.736347    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:21.814082 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.814048    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-xks-ca\") pod \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") "
Nov 02 16:12:21.814253 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.814181    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd64c27-4a21-4e3a-a514-81a2c9da944e-kube-api-access-zdrn9\" (UniqueName: \"kubernetes.io/projected/2dd64c27-4a21-4e3a-a514-81a2c9da944e-kube-api-access-zdrn9\") pod \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") "
Nov 02 16:12:21.814253 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.814206    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-openshift-ca\") pod \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\" (UID: \"2dd64c27-4a21-4e3a-a514-81a2c9da944e\") "
Nov 02 16:12:21.814631 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.814598    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-openshift-ca" pod "2dd64c27-4a21-4e3a-a514-81a2c9da944e" (UID: "2dd64c27-4a21-4e3a-a514-81a2c9da944e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:12:21.816146 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.816122    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-xks-ca" pod "2dd64c27-4a21-4e3a-a514-81a2c9da944e" (UID: "2dd64c27-4a21-4e3a-a514-81a2c9da944e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:12:21.816241 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.816186    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd64c27-4a21-4e3a-a514-81a2c9da944e-kube-api-access-zdrn9" pod "2dd64c27-4a21-4e3a-a514-81a2c9da944e" (UID: "2dd64c27-4a21-4e3a-a514-81a2c9da944e"). InnerVolumeSpecName "kube-api-access-zdrn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:12:21.915095 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.915062    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:12:21.915095 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.915089    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdrn9\" (UniqueName: \"kubernetes.io/projected/2dd64c27-4a21-4e3a-a514-81a2c9da944e-kube-api-access-zdrn9\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:12:21.915095 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.915099    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd64c27-4a21-4e3a-a514-81a2c9da944e-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:12:21.939413 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:21.939381    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:22.580921 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:22.580889    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"
Nov 02 16:12:22.580921 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:22.580906    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt" event={"ID":"2dd64c27-4a21-4e3a-a514-81a2c9da944e","Type":"ContainerDied","Data":"7a39ded1156d32cf6b4e9b078a2006d6733ecde23a14558fa9dd94e3e19da464"}
Nov 02 16:12:22.581211 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:22.580949    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a39ded1156d32cf6b4e9b078a2006d6733ecde23a14558fa9dd94e3e19da464"
Nov 02 16:12:22.705054 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:12:22.705016    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:12:23.420208 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:23.420166    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:25.198472 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:25.198431    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:26.996132 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:26.996095    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:27.148888 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:27.148849    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:28.791211 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:28.791133    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:29.103518 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:29.103432    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:29.255662 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:29.255613    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:30.502269 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:30.502234    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:37.072539 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:37.072494    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:37.579783 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:37.579750    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:37.738422 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:37.738386    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:37.741431 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:37.741407    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"]
Nov 02 16:12:37.753127 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:37.753103    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-a7b20c-nx2wt"]
Nov 02 16:12:37.860816 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:37.860792    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd64c27-4a21-4e3a-a514-81a2c9da944e" path="/var/lib/kubelet/pods/2dd64c27-4a21-4e3a-a514-81a2c9da944e/volumes"
Nov 02 16:12:38.246496 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:38.246403    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:40.240246 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:40.240203    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:12:44.031634 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:44.031592    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"]
Nov 02 16:12:44.044084 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:44.044049    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3dad175-mnshr"]
Nov 02 16:12:44.511593 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:44.511552    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:44.660460 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:44.660418    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:44.857730 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:44.857612    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:12:45.861232 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:45.861200    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="878261e4-6245-4a82-b67a-fe3fa159c4de" path="/var/lib/kubelet/pods/878261e4-6245-4a82-b67a-fe3fa159c4de/volumes"
Nov 02 16:12:47.460529 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:47.460494    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:12:52.705110 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:12:52.705075    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:12:55.751456 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:12:55.751418    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:13:02.393049 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.393009    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"]
Nov 02 16:13:02.393512 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.393495    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2dd64c27-4a21-4e3a-a514-81a2c9da944e" containerName="revoke-keys"
Nov 02 16:13:02.393565 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.393514    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd64c27-4a21-4e3a-a514-81a2c9da944e" containerName="revoke-keys"
Nov 02 16:13:02.393565 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.393528    7019 cpu_manager.go:414] "containerMap: removing container" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy"
Nov 02 16:13:02.393565 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.393533    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44ae50f-9b0f-40b4-af59-24170809ca56" containerName="istio-proxy"
Nov 02 16:13:02.393699 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.393617    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd64c27-4a21-4e3a-a514-81a2c9da944e" containerName="revoke-keys"
Nov 02 16:13:02.396759 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.396736    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.417452 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.417421    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"]
Nov 02 16:13:02.428597 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.428568    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:13:02.428597 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.428588    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8rvm8\""
Nov 02 16:13:02.491697 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.491639    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smlgg\" (UniqueName: \"kubernetes.io/projected/b6527197-d46e-4153-be1d-073079842549-kube-api-access-smlgg\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.491876 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.491823    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6527197-d46e-4153-be1d-073079842549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.491933 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.491902    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6527197-d46e-4153-be1d-073079842549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.592447 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.592404    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smlgg\" (UniqueName: \"kubernetes.io/projected/b6527197-d46e-4153-be1d-073079842549-kube-api-access-smlgg\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.592679 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.592492    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6527197-d46e-4153-be1d-073079842549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.592679 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.592532    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6527197-d46e-4153-be1d-073079842549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.593175 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.593154    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6527197-d46e-4153-be1d-073079842549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.594974 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.594952    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6527197-d46e-4153-be1d-073079842549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.616982 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.616956    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smlgg\" (UniqueName: \"kubernetes.io/projected/b6527197-d46e-4153-be1d-073079842549-kube-api-access-smlgg\") pod \"maas-api-revoke-sub-f03629db-8phrw\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:02.709772 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:02.709693    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:03.007782 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:03.007742    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"]
Nov 02 16:13:03.008111 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:13:03.008084    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6527197_d46e_4153_be1d_073079842549.slice/crio-47b14833d0c5a2a42d9ddd3372dd58a6a0de9f2ff84756f46651828d799a4f7f WatchSource:0}: Error finding container 47b14833d0c5a2a42d9ddd3372dd58a6a0de9f2ff84756f46651828d799a4f7f: Status 404 returned error can't find the container with id 47b14833d0c5a2a42d9ddd3372dd58a6a0de9f2ff84756f46651828d799a4f7f
Nov 02 16:13:03.834659 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:03.834611    7019 generic.go:356] "Generic (PLEG): container finished" podID="b6527197-d46e-4153-be1d-073079842549" containerID="cc5f551fe8832474ea29a2c218ec669b0d21cef8115b0f4256322ec28da3ee26" exitCode=0
Nov 02 16:13:03.835085 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:03.834682    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw" event={"ID":"b6527197-d46e-4153-be1d-073079842549","Type":"ContainerDied","Data":"cc5f551fe8832474ea29a2c218ec669b0d21cef8115b0f4256322ec28da3ee26"}
Nov 02 16:13:03.835085 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:03.834732    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw" event={"ID":"b6527197-d46e-4153-be1d-073079842549","Type":"ContainerStarted","Data":"47b14833d0c5a2a42d9ddd3372dd58a6a0de9f2ff84756f46651828d799a4f7f"}
Nov 02 16:13:04.973727 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:04.973695    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:05.016788 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.016756    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6527197-d46e-4153-be1d-073079842549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6527197-d46e-4153-be1d-073079842549-maas-api-xks-ca\") pod \"b6527197-d46e-4153-be1d-073079842549\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") "
Nov 02 16:13:05.017074 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.016820    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6527197-d46e-4153-be1d-073079842549-kube-api-access-smlgg\" (UniqueName: \"kubernetes.io/projected/b6527197-d46e-4153-be1d-073079842549-kube-api-access-smlgg\") pod \"b6527197-d46e-4153-be1d-073079842549\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") "
Nov 02 16:13:05.017074 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.016867    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6527197-d46e-4153-be1d-073079842549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6527197-d46e-4153-be1d-073079842549-maas-api-openshift-ca\") pod \"b6527197-d46e-4153-be1d-073079842549\" (UID: \"b6527197-d46e-4153-be1d-073079842549\") "
Nov 02 16:13:05.017551 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.017521    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6527197-d46e-4153-be1d-073079842549-maas-api-openshift-ca" pod "b6527197-d46e-4153-be1d-073079842549" (UID: "b6527197-d46e-4153-be1d-073079842549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:13:05.019408 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.019376    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6527197-d46e-4153-be1d-073079842549-maas-api-xks-ca" pod "b6527197-d46e-4153-be1d-073079842549" (UID: "b6527197-d46e-4153-be1d-073079842549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:13:05.019540 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.019381    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6527197-d46e-4153-be1d-073079842549-kube-api-access-smlgg" pod "b6527197-d46e-4153-be1d-073079842549" (UID: "b6527197-d46e-4153-be1d-073079842549"). InnerVolumeSpecName "kube-api-access-smlgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:13:05.118146 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.118119    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6527197-d46e-4153-be1d-073079842549-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:05.118146 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.118145    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smlgg\" (UniqueName: \"kubernetes.io/projected/b6527197-d46e-4153-be1d-073079842549-kube-api-access-smlgg\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:05.118332 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.118154    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6527197-d46e-4153-be1d-073079842549-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:05.846855 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.846827    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"
Nov 02 16:13:05.847026 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.846862    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw" event={"ID":"b6527197-d46e-4153-be1d-073079842549","Type":"ContainerDied","Data":"47b14833d0c5a2a42d9ddd3372dd58a6a0de9f2ff84756f46651828d799a4f7f"}
Nov 02 16:13:05.847026 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:05.846913    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47b14833d0c5a2a42d9ddd3372dd58a6a0de9f2ff84756f46651828d799a4f7f"
Nov 02 16:13:06.140220 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:06.140184    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:13:08.308528 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:08.308492    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:13:13.657940 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:13.657903    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:13:20.183209 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:20.183168    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:13:20.334775 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:20.334739    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:13:22.062236 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:22.062201    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:13:22.704333 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:13:22.704287    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:13:24.741349 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.741315    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"]
Nov 02 16:13:24.741976 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.741955    7019 cpu_manager.go:414] "containerMap: removing container" podUID="b6527197-d46e-4153-be1d-073079842549" containerName="revoke-keys"
Nov 02 16:13:24.742069 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.741978    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6527197-d46e-4153-be1d-073079842549" containerName="revoke-keys"
Nov 02 16:13:24.742140 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.742116    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6527197-d46e-4153-be1d-073079842549" containerName="revoke-keys"
Nov 02 16:13:24.745865 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.745835    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.762406 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.762378    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:13:24.775550 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.775519    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8rvm8\""
Nov 02 16:13:24.777163 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.777137    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"]
Nov 02 16:13:24.826729 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.826700    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.826893 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.826768    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.826893 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.826855    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbp65\" (UniqueName: \"kubernetes.io/projected/74f9a965-c0c4-435e-80ac-4d3a48c1c260-kube-api-access-hbp65\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.927540 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.927504    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.927737 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.927558    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbp65\" (UniqueName: \"kubernetes.io/projected/74f9a965-c0c4-435e-80ac-4d3a48c1c260-kube-api-access-hbp65\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.927737 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.927688    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.928368 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.928349    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.929962 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.929942    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:24.951389 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:24.951367    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbp65\" (UniqueName: \"kubernetes.io/projected/74f9a965-c0c4-435e-80ac-4d3a48c1c260-kube-api-access-hbp65\") pod \"maas-api-revoke-sub-0165d7d2-vqxj9\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:25.058967 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:25.058893    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:25.358261 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:13:25.358230    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f9a965_c0c4_435e_80ac_4d3a48c1c260.slice/crio-ccfcd5f42b39716a1ffb126423e217cdfc759141eda8823c0c86606bbd7f0c46 WatchSource:0}: Error finding container ccfcd5f42b39716a1ffb126423e217cdfc759141eda8823c0c86606bbd7f0c46: Status 404 returned error can't find the container with id ccfcd5f42b39716a1ffb126423e217cdfc759141eda8823c0c86606bbd7f0c46
Nov 02 16:13:25.358798 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:25.358771    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"]
Nov 02 16:13:25.973155 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:25.973077    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9" event={"ID":"74f9a965-c0c4-435e-80ac-4d3a48c1c260","Type":"ContainerStarted","Data":"727f41fdf21596f592a650cf879b8d6a925fe0eef17352ef22b6904730876ab8"}
Nov 02 16:13:25.973155 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:25.973126    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9" event={"ID":"74f9a965-c0c4-435e-80ac-4d3a48c1c260","Type":"ContainerStarted","Data":"ccfcd5f42b39716a1ffb126423e217cdfc759141eda8823c0c86606bbd7f0c46"}
Nov 02 16:13:26.015450 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:26.015408    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9" podStartSLOduration=2.015395114 podStartE2EDuration="2.015395114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:13:26.014878239 +0000 UTC m=+4600.268831863" watchObservedRunningTime="2026-11-02 16:13:26.015395114 +0000 UTC m=+4600.269348732"
Nov 02 16:13:26.979957 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:26.979874    7019 generic.go:356] "Generic (PLEG): container finished" podID="74f9a965-c0c4-435e-80ac-4d3a48c1c260" containerID="727f41fdf21596f592a650cf879b8d6a925fe0eef17352ef22b6904730876ab8" exitCode=0
Nov 02 16:13:26.979957 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:26.979923    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9" event={"ID":"74f9a965-c0c4-435e-80ac-4d3a48c1c260","Type":"ContainerDied","Data":"727f41fdf21596f592a650cf879b8d6a925fe0eef17352ef22b6904730876ab8"}
Nov 02 16:13:27.186092 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:27.186045    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:13:28.126539 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.126516    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:28.257471 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.257396    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f9a965-c0c4-435e-80ac-4d3a48c1c260-kube-api-access-hbp65\" (UniqueName: \"kubernetes.io/projected/74f9a965-c0c4-435e-80ac-4d3a48c1c260-kube-api-access-hbp65\") pod \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") "
Nov 02 16:13:28.257639 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.257499    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-xks-ca\") pod \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") "
Nov 02 16:13:28.257639 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.257539    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-openshift-ca\") pod \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\" (UID: \"74f9a965-c0c4-435e-80ac-4d3a48c1c260\") "
Nov 02 16:13:28.257982 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.257957    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-openshift-ca" pod "74f9a965-c0c4-435e-80ac-4d3a48c1c260" (UID: "74f9a965-c0c4-435e-80ac-4d3a48c1c260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:13:28.259594 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.259564    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-xks-ca" pod "74f9a965-c0c4-435e-80ac-4d3a48c1c260" (UID: "74f9a965-c0c4-435e-80ac-4d3a48c1c260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:13:28.259734 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.259709    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f9a965-c0c4-435e-80ac-4d3a48c1c260-kube-api-access-hbp65" pod "74f9a965-c0c4-435e-80ac-4d3a48c1c260" (UID: "74f9a965-c0c4-435e-80ac-4d3a48c1c260"). InnerVolumeSpecName "kube-api-access-hbp65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:13:28.358877 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.358840    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbp65\" (UniqueName: \"kubernetes.io/projected/74f9a965-c0c4-435e-80ac-4d3a48c1c260-kube-api-access-hbp65\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:28.358877 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.358872    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:28.359066 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.358886    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74f9a965-c0c4-435e-80ac-4d3a48c1c260-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:28.991982 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.991949    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9"
Nov 02 16:13:28.992173 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.991980    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vqxj9" event={"ID":"74f9a965-c0c4-435e-80ac-4d3a48c1c260","Type":"ContainerDied","Data":"ccfcd5f42b39716a1ffb126423e217cdfc759141eda8823c0c86606bbd7f0c46"}
Nov 02 16:13:28.992173 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:28.992032    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccfcd5f42b39716a1ffb126423e217cdfc759141eda8823c0c86606bbd7f0c46"
Nov 02 16:13:35.681511 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.681458    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"]
Nov 02 16:13:35.682190 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.682168    7019 cpu_manager.go:414] "containerMap: removing container" podUID="74f9a965-c0c4-435e-80ac-4d3a48c1c260" containerName="revoke-keys"
Nov 02 16:13:35.682190 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.682190    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f9a965-c0c4-435e-80ac-4d3a48c1c260" containerName="revoke-keys"
Nov 02 16:13:35.682345 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.682275    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f9a965-c0c4-435e-80ac-4d3a48c1c260" containerName="revoke-keys"
Nov 02 16:13:35.686123 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.686102    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.706432 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.706404    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"]
Nov 02 16:13:35.716902 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.716875    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8rvm8\""
Nov 02 16:13:35.717049 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.716929    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 16:13:35.835268 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.835228    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.835463 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.835283    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.835463 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.835442    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfx4t\" (UniqueName: \"kubernetes.io/projected/20c14193-5b1d-4961-b5e6-021caf6a3e6f-kube-api-access-lfx4t\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.936600 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.936505    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfx4t\" (UniqueName: \"kubernetes.io/projected/20c14193-5b1d-4961-b5e6-021caf6a3e6f-kube-api-access-lfx4t\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.936600 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.936572    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.936859 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.936623    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.937257 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.937228    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.939028 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.939008    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.961175 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.961145    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfx4t\" (UniqueName: \"kubernetes.io/projected/20c14193-5b1d-4961-b5e6-021caf6a3e6f-kube-api-access-lfx4t\") pod \"maas-api-revoke-sub-6b19338f-cth4b\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:35.999565 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:35.999533    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:36.312944 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:36.312907    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"]
Nov 02 16:13:36.313084 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:13:36.313016    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c14193_5b1d_4961_b5e6_021caf6a3e6f.slice/crio-0f4f9f2014a173ef061bf1cfee4b5695c691d16ab10d9a6d32873a113db37546 WatchSource:0}: Error finding container 0f4f9f2014a173ef061bf1cfee4b5695c691d16ab10d9a6d32873a113db37546: Status 404 returned error can't find the container with id 0f4f9f2014a173ef061bf1cfee4b5695c691d16ab10d9a6d32873a113db37546
Nov 02 16:13:37.038416 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:37.038369    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b" event={"ID":"20c14193-5b1d-4961-b5e6-021caf6a3e6f","Type":"ContainerStarted","Data":"db94184173bfa209f2e4441774f5dd8ca1103c0d5f4ae0bffd763de115a42153"}
Nov 02 16:13:37.038894 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:37.038444    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b" event={"ID":"20c14193-5b1d-4961-b5e6-021caf6a3e6f","Type":"ContainerStarted","Data":"0f4f9f2014a173ef061bf1cfee4b5695c691d16ab10d9a6d32873a113db37546"}
Nov 02 16:13:37.081685 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:37.081616    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b" podStartSLOduration=2.08160207 podStartE2EDuration="2.08160207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:13:37.081310317 +0000 UTC m=+4611.335263942" watchObservedRunningTime="2026-11-02 16:13:37.08160207 +0000 UTC m=+4611.335555695"
Nov 02 16:13:38.044375 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:38.044339    7019 generic.go:356] "Generic (PLEG): container finished" podID="20c14193-5b1d-4961-b5e6-021caf6a3e6f" containerID="db94184173bfa209f2e4441774f5dd8ca1103c0d5f4ae0bffd763de115a42153" exitCode=0
Nov 02 16:13:38.044800 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:38.044439    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b" event={"ID":"20c14193-5b1d-4961-b5e6-021caf6a3e6f","Type":"ContainerDied","Data":"db94184173bfa209f2e4441774f5dd8ca1103c0d5f4ae0bffd763de115a42153"}
Nov 02 16:13:39.184705 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.184681    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:39.270257 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.270223    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-xks-ca\") pod \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") "
Nov 02 16:13:39.270433 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.270279    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-openshift-ca\") pod \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") "
Nov 02 16:13:39.270433 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.270387    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c14193-5b1d-4961-b5e6-021caf6a3e6f-kube-api-access-lfx4t\" (UniqueName: \"kubernetes.io/projected/20c14193-5b1d-4961-b5e6-021caf6a3e6f-kube-api-access-lfx4t\") pod \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\" (UID: \"20c14193-5b1d-4961-b5e6-021caf6a3e6f\") "
Nov 02 16:13:39.270760 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.270732    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-openshift-ca" pod "20c14193-5b1d-4961-b5e6-021caf6a3e6f" (UID: "20c14193-5b1d-4961-b5e6-021caf6a3e6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:13:39.272307 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.272275    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-xks-ca" pod "20c14193-5b1d-4961-b5e6-021caf6a3e6f" (UID: "20c14193-5b1d-4961-b5e6-021caf6a3e6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 16:13:39.272479 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.272455    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c14193-5b1d-4961-b5e6-021caf6a3e6f-kube-api-access-lfx4t" pod "20c14193-5b1d-4961-b5e6-021caf6a3e6f" (UID: "20c14193-5b1d-4961-b5e6-021caf6a3e6f"). InnerVolumeSpecName "kube-api-access-lfx4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:13:39.372078 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.372043    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfx4t\" (UniqueName: \"kubernetes.io/projected/20c14193-5b1d-4961-b5e6-021caf6a3e6f-kube-api-access-lfx4t\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:39.372078 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.372073    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-xks-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:39.372266 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:39.372089    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20c14193-5b1d-4961-b5e6-021caf6a3e6f-maas-api-openshift-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:13:40.055676 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:40.055633    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b"
Nov 02 16:13:40.055844 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:40.055633    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-cth4b" event={"ID":"20c14193-5b1d-4961-b5e6-021caf6a3e6f","Type":"ContainerDied","Data":"0f4f9f2014a173ef061bf1cfee4b5695c691d16ab10d9a6d32873a113db37546"}
Nov 02 16:13:40.055844 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:40.055765    7019 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f4f9f2014a173ef061bf1cfee4b5695c691d16ab10d9a6d32873a113db37546"
Nov 02 16:13:47.048381 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:47.048346    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-tr9hf"]
Nov 02 16:13:52.081997 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:13:52.081963    7019 scope.go:122] "RemoveContainer" containerID="1193798591f4ed38e6f95767f87540e56cac901c956841b33e0da60f52f17c92"
Nov 02 16:13:52.704028 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:13:52.703989    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:14:08.416138 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:08.416106    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:14:15.366067 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:15.366034    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:14:22.183663 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:22.183607    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:14:22.704044 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:14:22.704008    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:14:27.320367 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:27.320330    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:14:39.577693 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:39.577638    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6bwg2_5d8e761f-f938-4be7-97c2-bbc0e34045d5/manager/0.log"
Nov 02 16:14:40.076373 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:40.076339    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lqzvk_6f0dc694-6e68-4cb1-a298-2be54b98a7b0/manager/0.log"
Nov 02 16:14:40.607465 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:40.607435    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hh45n_d4267139-20cc-4984-988b-bbd1082c74b1/server/0.log"
Nov 02 16:14:41.032486 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:41.032456    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ngklw_9801b861-5043-4810-96b3-e96603bd9e47/proxy/0.log"
Nov 02 16:14:42.224055 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:42.224023    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vqxj9_74f9a965-c0c4-435e-80ac-4d3a48c1c260/revoke-keys/0.log"
Nov 02 16:14:42.628024 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:42.627988    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cth4b_20c14193-5b1d-4961-b5e6-021caf6a3e6f/revoke-keys/0.log"
Nov 02 16:14:42.809406 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:42.809374    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8phrw_b6527197-d46e-4153-be1d-073079842549/revoke-keys/0.log"
Nov 02 16:14:42.993027 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:42.992954    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jc2wz_0d405b00-cf02-4e4c-ae08-53b0f5ab5980/postgres/0.log"
Nov 02 16:14:46.531817 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:46.531771    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:14:52.102930 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:52.102897    7019 scope.go:122] "RemoveContainer" containerID="8900954e15fcaf188371062a2cf175d131bb1e21556356bb3b1210f3f834bb91"
Nov 02 16:14:52.703900 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:14:52.703858    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:14:54.845899 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:54.845862    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-s4kqd_3afbbd3a-0443-4878-8556-172bc7d3e8e0/manager/0.log"
Nov 02 16:14:55.484847 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:55.484816    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g4sbf_952f1fec-5220-4de1-8107-9de8bb4d7fea/manager/0.log"
Nov 02 16:14:55.759414 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:55.759331    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-tr9hf_74a06875-fd82-45ab-b150-cffc05814608/limitador/0.log"
Nov 02 16:14:55.966301 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:55.966265    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8b92j_d457cf34-abad-45c0-884f-871d30378f1b/manager/0.log"
Nov 02 16:14:56.806163 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:56.806126    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cjrwd_ee1d3ebd-ce9d-467d-8d29-0f09eafc306b/istio-proxy/0.log"
Nov 02 16:14:57.829262 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:57.829228    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8l7bs_74ffc3f9-8341-41be-8b15-c9829dfb6fb0/istio-proxy/0.log"
Nov 02 16:14:58.049075 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:58.049047    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-cq5h5_30ae8e5d-46c2-4af1-9ea4-e27446758848/istio-proxy/0.log"
Nov 02 16:14:59.459760 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:59.459730    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-xqbf4_d1affeef-d790-45c6-b74c-9053b4f08f57/istio-proxy/0.log"
Nov 02 16:14:59.676839 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:14:59.676806    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl_d7147f7a-95de-4a28-939a-5b0eb44a22cc/istio-proxy/0.log"
Nov 02 16:15:00.163417 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.163381    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"]
Nov 02 16:15:00.163872 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.163857    7019 cpu_manager.go:414] "containerMap: removing container" podUID="20c14193-5b1d-4961-b5e6-021caf6a3e6f" containerName="revoke-keys"
Nov 02 16:15:00.163922 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.163873    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c14193-5b1d-4961-b5e6-021caf6a3e6f" containerName="revoke-keys"
Nov 02 16:15:00.163971 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.163961    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c14193-5b1d-4961-b5e6-021caf6a3e6f" containerName="revoke-keys"
Nov 02 16:15:00.167231 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.167212    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.178966 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.178936    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5dgc\" (UniqueName: \"kubernetes.io/projected/2c8a4d87-6f5f-4853-9978-6841e3f5e910-kube-api-access-q5dgc\") pod \"maas-api-key-cleanup-29893935-ghcvw\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.179068 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.179002    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a4d87-6f5f-4853-9978-6841e3f5e910-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893935-ghcvw\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.187062 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.187031    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"]
Nov 02 16:15:00.198197 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.198174    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8rvm8\""
Nov 02 16:15:00.279732 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.279695    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5dgc\" (UniqueName: \"kubernetes.io/projected/2c8a4d87-6f5f-4853-9978-6841e3f5e910-kube-api-access-q5dgc\") pod \"maas-api-key-cleanup-29893935-ghcvw\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.279732 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.279734    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a4d87-6f5f-4853-9978-6841e3f5e910-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893935-ghcvw\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.280389 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.280366    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a4d87-6f5f-4853-9978-6841e3f5e910-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893935-ghcvw\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.304687 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.304631    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5dgc\" (UniqueName: \"kubernetes.io/projected/2c8a4d87-6f5f-4853-9978-6841e3f5e910-kube-api-access-q5dgc\") pod \"maas-api-key-cleanup-29893935-ghcvw\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.485333 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.485241    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:00.783248 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:00.783202    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"]
Nov 02 16:15:00.783532 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:15:00.783499    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8a4d87_6f5f_4853_9978_6841e3f5e910.slice/crio-13e2a2f069e05aa8e759ca71767ee7bd4870cbcf7d8271ccd24f731f3b92e4ee WatchSource:0}: Error finding container 13e2a2f069e05aa8e759ca71767ee7bd4870cbcf7d8271ccd24f731f3b92e4ee: Status 404 returned error can't find the container with id 13e2a2f069e05aa8e759ca71767ee7bd4870cbcf7d8271ccd24f731f3b92e4ee
Nov 02 16:15:01.495720 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:01.495685    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" event={"ID":"2c8a4d87-6f5f-4853-9978-6841e3f5e910","Type":"ContainerStarted","Data":"ff7b70702243a5fac05ee686f103861565f9f3cf88a8a8d354451d56fd2b979a"}
Nov 02 16:15:01.496095 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:01.495745    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" event={"ID":"2c8a4d87-6f5f-4853-9978-6841e3f5e910","Type":"ContainerStarted","Data":"13e2a2f069e05aa8e759ca71767ee7bd4870cbcf7d8271ccd24f731f3b92e4ee"}
Nov 02 16:15:02.502880 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:02.502845    7019 generic.go:356] "Generic (PLEG): container finished" podID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerID="ff7b70702243a5fac05ee686f103861565f9f3cf88a8a8d354451d56fd2b979a" exitCode=60
Nov 02 16:15:02.503282 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:02.502935    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" event={"ID":"2c8a4d87-6f5f-4853-9978-6841e3f5e910","Type":"ContainerDied","Data":"ff7b70702243a5fac05ee686f103861565f9f3cf88a8a8d354451d56fd2b979a"}
Nov 02 16:15:02.503282 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:02.503127    7019 scope.go:122] "RemoveContainer" containerID="ff7b70702243a5fac05ee686f103861565f9f3cf88a8a8d354451d56fd2b979a"
Nov 02 16:15:03.510055 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:03.510018    7019 generic.go:356] "Generic (PLEG): container finished" podID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerID="f675f1879f4b0ba345974b4f646d4ce513a21fe85e7858716e5c1510fc7d32f5" exitCode=60
Nov 02 16:15:03.510557 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:03.510062    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" event={"ID":"2c8a4d87-6f5f-4853-9978-6841e3f5e910","Type":"ContainerDied","Data":"f675f1879f4b0ba345974b4f646d4ce513a21fe85e7858716e5c1510fc7d32f5"}
Nov 02 16:15:03.510557 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:03.510115    7019 scope.go:122] "RemoveContainer" containerID="ff7b70702243a5fac05ee686f103861565f9f3cf88a8a8d354451d56fd2b979a"
Nov 02 16:15:03.510557 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:03.510417    7019 scope.go:122] "RemoveContainer" containerID="f675f1879f4b0ba345974b4f646d4ce513a21fe85e7858716e5c1510fc7d32f5"
Nov 02 16:15:03.510811 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:15:03.510790    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893935-ghcvw_odh-ai-gateway-infra(2c8a4d87-6f5f-4853-9978-6841e3f5e910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910"
Nov 02 16:15:04.516833 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:04.516806    7019 scope.go:122] "RemoveContainer" containerID="f675f1879f4b0ba345974b4f646d4ce513a21fe85e7858716e5c1510fc7d32f5"
Nov 02 16:15:04.517237 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:15:04.517040    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893935-ghcvw_odh-ai-gateway-infra(2c8a4d87-6f5f-4853-9978-6841e3f5e910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910"
Nov 02 16:15:08.531633 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:08.531594    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:15:14.395211 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:14.395147    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6bwg2_5d8e761f-f938-4be7-97c2-bbc0e34045d5/manager/0.log"
Nov 02 16:15:14.814478 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:14.814397    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-lqzvk_6f0dc694-6e68-4cb1-a298-2be54b98a7b0/manager/0.log"
Nov 02 16:15:15.134783 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:15.134748    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:15:15.259660 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:15.259629    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hh45n_d4267139-20cc-4984-988b-bbd1082c74b1/server/0.log"
Nov 02 16:15:15.286213 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:15.286187    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:15:15.678279 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:15.678249    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-ngklw_9801b861-5043-4810-96b3-e96603bd9e47/proxy/0.log"
Nov 02 16:15:16.659753 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:16.659723    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29893935-ghcvw_2c8a4d87-6f5f-4853-9978-6841e3f5e910/cleanup/1.log"
Nov 02 16:15:16.842816 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:16.842784    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vqxj9_74f9a965-c0c4-435e-80ac-4d3a48c1c260/revoke-keys/0.log"
Nov 02 16:15:17.030260 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:17.030181    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-cth4b_20c14193-5b1d-4961-b5e6-021caf6a3e6f/revoke-keys/0.log"
Nov 02 16:15:17.209406 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:17.209377    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-8phrw_b6527197-d46e-4153-be1d-073079842549/revoke-keys/0.log"
Nov 02 16:15:17.402688 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:17.402618    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jc2wz_0d405b00-cf02-4e4c-ae08-53b0f5ab5980/postgres/0.log"
Nov 02 16:15:17.854822 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:17.854729    7019 scope.go:122] "RemoveContainer" containerID="f675f1879f4b0ba345974b4f646d4ce513a21fe85e7858716e5c1510fc7d32f5"
Nov 02 16:15:18.588375 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:18.588293    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" event={"ID":"2c8a4d87-6f5f-4853-9978-6841e3f5e910","Type":"ContainerStarted","Data":"7a81391f629880415eb2c990d008d0ef645bf3ff5292619e6cf9363ed4043207"}
Nov 02 16:15:18.630171 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:18.630123    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" podStartSLOduration=18.63011104 podStartE2EDuration="18.63011104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:15:01.537633044 +0000 UTC m=+4695.791586667" watchObservedRunningTime="2026-11-02 16:15:18.63011104 +0000 UTC m=+4712.884064657"
Nov 02 16:15:18.856583 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:18.856438    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-s4kqd_3afbbd3a-0443-4878-8556-172bc7d3e8e0/manager/0.log"
Nov 02 16:15:19.430211 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.430175    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g4sbf_952f1fec-5220-4de1-8107-9de8bb4d7fea/manager/0.log"
Nov 02 16:15:19.595819 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.595789    7019 generic.go:356] "Generic (PLEG): container finished" podID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerID="7a81391f629880415eb2c990d008d0ef645bf3ff5292619e6cf9363ed4043207" exitCode=60
Nov 02 16:15:19.595996 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.595864    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" event={"ID":"2c8a4d87-6f5f-4853-9978-6841e3f5e910","Type":"ContainerDied","Data":"7a81391f629880415eb2c990d008d0ef645bf3ff5292619e6cf9363ed4043207"}
Nov 02 16:15:19.595996 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.595923    7019 scope.go:122] "RemoveContainer" containerID="f675f1879f4b0ba345974b4f646d4ce513a21fe85e7858716e5c1510fc7d32f5"
Nov 02 16:15:19.596278 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.596259    7019 scope.go:122] "RemoveContainer" containerID="7a81391f629880415eb2c990d008d0ef645bf3ff5292619e6cf9363ed4043207"
Nov 02 16:15:19.596507 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:15:19.596488    7019 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893935-ghcvw_odh-ai-gateway-infra(2c8a4d87-6f5f-4853-9978-6841e3f5e910)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910"
Nov 02 16:15:19.647670 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.647626    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"]
Nov 02 16:15:19.649283 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.649264    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-tr9hf_74a06875-fd82-45ab-b150-cffc05814608/limitador/0.log"
Nov 02 16:15:19.838388 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:19.838296    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-8b92j_d457cf34-abad-45c0-884f-871d30378f1b/manager/0.log"
Nov 02 16:15:20.386697 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.386640    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cjrwd_ee1d3ebd-ce9d-467d-8d29-0f09eafc306b/istio-proxy/0.log"
Nov 02 16:15:20.737711 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.737689    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:20.881851 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.881817    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8a4d87-6f5f-4853-9978-6841e3f5e910-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a4d87-6f5f-4853-9978-6841e3f5e910-maas-api-service-ca\") pod \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") "
Nov 02 16:15:20.882031 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.881954    7019 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8a4d87-6f5f-4853-9978-6841e3f5e910-kube-api-access-q5dgc\" (UniqueName: \"kubernetes.io/projected/2c8a4d87-6f5f-4853-9978-6841e3f5e910-kube-api-access-q5dgc\") pod \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\" (UID: \"2c8a4d87-6f5f-4853-9978-6841e3f5e910\") "
Nov 02 16:15:20.882236 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.882208    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8a4d87-6f5f-4853-9978-6841e3f5e910-maas-api-service-ca" pod "2c8a4d87-6f5f-4853-9978-6841e3f5e910" (UID: "2c8a4d87-6f5f-4853-9978-6841e3f5e910"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 16:15:20.883935 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.883908    7019 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8a4d87-6f5f-4853-9978-6841e3f5e910-kube-api-access-q5dgc" pod "2c8a4d87-6f5f-4853-9978-6841e3f5e910" (UID: "2c8a4d87-6f5f-4853-9978-6841e3f5e910"). InnerVolumeSpecName "kube-api-access-q5dgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 16:15:20.983224 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.983157    7019 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a4d87-6f5f-4853-9978-6841e3f5e910-maas-api-service-ca\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:15:20.983224 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:20.983181    7019 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5dgc\" (UniqueName: \"kubernetes.io/projected/2c8a4d87-6f5f-4853-9978-6841e3f5e910-kube-api-access-q5dgc\") on node \"ip-10-0-129-45.ec2.internal\" DevicePath \"\""
Nov 02 16:15:21.424370 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.424333    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8l7bs_74ffc3f9-8341-41be-8b15-c9829dfb6fb0/istio-proxy/0.log"
Nov 02 16:15:21.608936 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.608904    7019 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"
Nov 02 16:15:21.608936 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.608923    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw" event={"ID":"2c8a4d87-6f5f-4853-9978-6841e3f5e910","Type":"ContainerDied","Data":"13e2a2f069e05aa8e759ca71767ee7bd4870cbcf7d8271ccd24f731f3b92e4ee"}
Nov 02 16:15:21.609183 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.609013    7019 scope.go:122] "RemoveContainer" containerID="7a81391f629880415eb2c990d008d0ef645bf3ff5292619e6cf9363ed4043207"
Nov 02 16:15:21.629867 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.629843    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-cq5h5_30ae8e5d-46c2-4af1-9ea4-e27446758848/istio-proxy/0.log"
Nov 02 16:15:21.661836 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.661805    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"]
Nov 02 16:15:21.674829 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.674770    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893935-ghcvw"]
Nov 02 16:15:21.860577 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:21.860546    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" path="/var/lib/kubelet/pods/2c8a4d87-6f5f-4853-9978-6841e3f5e910/volumes"
Nov 02 16:15:22.317080 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:22.317043    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:15:22.704768 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:15:22.704732    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:15:23.026240 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:23.026164    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-xqbf4_d1affeef-d790-45c6-b74c-9053b4f08f57/istio-proxy/0.log"
Nov 02 16:15:23.227836 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:23.227791    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl_d7147f7a-95de-4a28-939a-5b0eb44a22cc/istio-proxy/0.log"
Nov 02 16:15:27.434032 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:15:27.433992    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:15:52.705238 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:15:52.705199    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:16:08.661396 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:16:08.661357    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:16:11.097561 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:16:11.097518    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:16:11.245094 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:16:11.245058    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:16:22.472455 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:16:22.472423    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:16:22.706383 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:16:22.706337    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:16:27.563476 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:16:27.563391    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:16:52.141661 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:16:52.141626    7019 scope.go:122] "RemoveContainer" containerID="3dd400e590d882ad502727eda56637920bc83813f0b788a61c84700722276070"
Nov 02 16:16:52.704409 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:16:52.704366    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:17:08.813521 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:08.813478    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:17:22.581632 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:22.581593    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:17:22.704106 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:17:22.704071    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:17:27.054972 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:27.054932    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:17:27.303082 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:27.303039    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:17:27.670979 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:27.670945    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:17:31.461128 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:31.461096    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l52v6_35254468-9d2e-4c69-a162-02ae0854b07f/global-pull-secret-syncer/0.log"
Nov 02 16:17:32.225002 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:32.224970    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log"
Nov 02 16:17:32.575098 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:32.575013    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-45.ec2.internal_cf4d75b8d1582cef207a6bcc13a46c78/haproxy/0.log"
Nov 02 16:17:49.739624 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.739580    7019 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"]
Nov 02 16:17:49.740271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.740243    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerName="cleanup"
Nov 02 16:17:49.740271 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.740265    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerName="cleanup"
Nov 02 16:17:49.740412 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.740308    7019 cpu_manager.go:414] "containerMap: removing container" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerName="cleanup"
Nov 02 16:17:49.740412 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.740317    7019 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerName="cleanup"
Nov 02 16:17:49.740526 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.740424    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerName="cleanup"
Nov 02 16:17:49.740526 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.740444    7019 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8a4d87-6f5f-4853-9978-6841e3f5e910" containerName="cleanup"
Nov 02 16:17:49.743872 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.743850    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.761311 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.761285    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xl725\"/\"kube-root-ca.crt\""
Nov 02 16:17:49.763046 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.763019    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"]
Nov 02 16:17:49.775988 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.775966    7019 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xl725\"/\"openshift-service-ca.crt\""
Nov 02 16:17:49.776111 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.776037    7019 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xl725\"/\"default-dockercfg-vkz4g\""
Nov 02 16:17:49.837141 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.837107    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-proc\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.837296 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.837191    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xltl\" (UniqueName: \"kubernetes.io/projected/1290c3d3-7752-4382-840b-c08101381a4d-kube-api-access-4xltl\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.837296 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.837258    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-podres\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.837296 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.837284    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-lib-modules\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.837431 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.837307    7019 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-sys\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.937946 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.937908    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-proc\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938130 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.937992    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xltl\" (UniqueName: \"kubernetes.io/projected/1290c3d3-7752-4382-840b-c08101381a4d-kube-api-access-4xltl\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938130 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.938018    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-podres\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938130 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.938032    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-proc\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938130 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.938044    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-lib-modules\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938130 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.938106    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-podres\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938398 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.938145    7019 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-sys\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938398 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.938202    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-lib-modules\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.938398 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.938251    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1290c3d3-7752-4382-840b-c08101381a4d-sys\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:49.963822 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:49.963793    7019 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xltl\" (UniqueName: \"kubernetes.io/projected/1290c3d3-7752-4382-840b-c08101381a4d-kube-api-access-4xltl\") pod \"perf-node-gather-daemonset-c4fmz\" (UID: \"1290c3d3-7752-4382-840b-c08101381a4d\") " pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:50.057349 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:50.057272    7019 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:50.357622 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:50.357592    7019 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"]
Nov 02 16:17:50.358215 ip-10-0-129-45 kubenswrapper[7019]: W1102 16:17:50.358189    7019 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1290c3d3_7752_4382_840b_c08101381a4d.slice/crio-0668b5abbe560ef5c512fadaea28e0064f248a08f82512f58999b232d8a2f71a WatchSource:0}: Error finding container 0668b5abbe560ef5c512fadaea28e0064f248a08f82512f58999b232d8a2f71a: Status 404 returned error can't find the container with id 0668b5abbe560ef5c512fadaea28e0064f248a08f82512f58999b232d8a2f71a
Nov 02 16:17:50.376189 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:50.376162    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz" event={"ID":"1290c3d3-7752-4382-840b-c08101381a4d","Type":"ContainerStarted","Data":"0668b5abbe560ef5c512fadaea28e0064f248a08f82512f58999b232d8a2f71a"}
Nov 02 16:17:51.383204 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:51.383164    7019 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz" event={"ID":"1290c3d3-7752-4382-840b-c08101381a4d","Type":"ContainerStarted","Data":"546917bdd27e3ba0893426737ef49187e24e85ad9a7d1c1d24dd97aec1046e8d"}
Nov 02 16:17:51.383639 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:51.383240    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:17:51.430617 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:51.430571    7019 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz" podStartSLOduration=2.430558865 podStartE2EDuration="2.430558865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:17:51.428806216 +0000 UTC m=+4865.682759840" watchObservedRunningTime="2026-11-02 16:17:51.430558865 +0000 UTC m=+4865.684512488"
Nov 02 16:17:52.704349 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:17:52.704307    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:17:57.398890 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:17:57.398815    7019 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xl725/perf-node-gather-daemonset-c4fmz"
Nov 02 16:18:06.034115 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:06.034079    7019 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"]
Nov 02 16:18:06.048503 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:06.048476    7019 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-8phrw"]
Nov 02 16:18:07.860182 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:07.860151    7019 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6527197-d46e-4153-be1d-073079842549" path="/var/lib/kubelet/pods/b6527197-d46e-4153-be1d-073079842549/volumes"
Nov 02 16:18:08.910938 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:08.910904    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024
Nov 02 16:18:11.454346 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:11.454312    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7c689579c-cjrwd_ee1d3ebd-ce9d-467d-8d29-0f09eafc306b/istio-proxy/0.log"
Nov 02 16:18:16.727918 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:16.727885    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-8l7bs_74ffc3f9-8341-41be-8b15-c9829dfb6fb0/istio-proxy/0.log"
Nov 02 16:18:16.792184 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:16.792153    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-cq5h5_30ae8e5d-46c2-4af1-9ea4-e27446758848/istio-proxy/0.log"
Nov 02 16:18:17.242513 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:17.242480    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fcb75c45f-xqbf4_d1affeef-d790-45c6-b74c-9053b4f08f57/istio-proxy/0.log"
Nov 02 16:18:17.304836 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:17.304806    7019 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-74987c9b47-8k2wl_d7147f7a-95de-4a28-939a-5b0eb44a22cc/istio-proxy/0.log"
Nov 02 16:18:22.704585 ip-10-0-129-45 kubenswrapper[7019]: E1102 16:18:22.704537    7019 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jzm4s" containerName="extractor"
Nov 02 16:18:22.724551 ip-10-0-129-45 kubenswrapper[7019]: I1102 16:18:22.724519    7019 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-29z4k_4ab14eae-d5f0-46e3-abda-c817d0e8b66e/konnectivity-agent/0.log" limit=1024